[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:07 dist-test-slave-c5d6 krb5kdc[8373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:07 dist-test-slave-c5d6 krb5kdc[8373](info): set up 2 sockets
Sep 25 15:15:07 dist-test-slave-c5d6 krb5kdc[8373](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 15:15:08.705368 8366 mini_kdc.cc:121] Time spent starting KDC: real 2.080s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:08 dist-test-slave-c5d6 krb5kdc[8373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:08.824019 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:36425
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:36425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:08.907586 8386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:08.907802 8386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:08.907827 8386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:08.908967 8386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:08.909008 8386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:08.909021 8386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:08.909034 8386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:08.909046 8386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:08.910537 8386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:36425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:36425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:08.910766 8386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:08.911003 8386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:08.916695 8392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:08.916899 8391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:08.916990 8386 server_base.cc:1034] running on GCE node
W20260925 15:15:08.917163 8394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:08.917478 8386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:08.917917 8386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:08.919122 8386 hybrid_clock.cc:648] HybridClock initialized: now 1790349308919106 us; error 33 us; skew 500 ppm
Sep 25 15:15:08 dist-test-slave-c5d6 krb5kdc[8373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:08.921905 8386 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:08.922525 8386 webserver.cc:460] Webserver started at http://127.8.43.190:45917/ using document root <none> and password file <none>
I20260925 15:15:08.922848 8386 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:08.922932 8386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:08.923126 8386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:08.924811 8386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "2bdcc30b5b404963bb6305e71c2535c4"
format_stamp: "Formatted at 2026-09-25 15:15:08 on dist-test-slave-c5d6"
server_key: "b7ff2343ca60a1b743bbe2f6b3fd6aad"
server_key_iv: "c323fc2d1af88c7d108a7e5a33550cc4"
server_key_version: "encryptionkey@0"
I20260925 15:15:08.925482 8386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "2bdcc30b5b404963bb6305e71c2535c4"
format_stamp: "Formatted at 2026-09-25 15:15:08 on dist-test-slave-c5d6"
server_key: "b7ff2343ca60a1b743bbe2f6b3fd6aad"
server_key_iv: "c323fc2d1af88c7d108a7e5a33550cc4"
server_key_version: "encryptionkey@0"
I20260925 15:15:08.927433 8386 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:15:08.928452 8401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:08.928720 8386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:08.928838 8386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "2bdcc30b5b404963bb6305e71c2535c4"
format_stamp: "Formatted at 2026-09-25 15:15:08 on dist-test-slave-c5d6"
server_key: "b7ff2343ca60a1b743bbe2f6b3fd6aad"
server_key_iv: "c323fc2d1af88c7d108a7e5a33550cc4"
server_key_version: "encryptionkey@0"
I20260925 15:15:08.928960 8386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:08.948083 8386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:08.950471 8386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:08.950635 8386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:08.955034 8386 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:36425
I20260925 15:15:08.955183 8453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:36425 every 8 connection(s)
I20260925 15:15:08.955600 8386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:08.956692 8454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:08.959010 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 8386
I20260925 15:15:08.959121 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:08.960019 8366 external_mini_cluster.cc:1428] Setting key 9dd50969e04a8b9d6991c8dc99d74087
I20260925 15:15:08.960052 8454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4: Bootstrap starting.
I20260925 15:15:08.961198 8454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:08.961531 8454 log.cc:824] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:08.962308 8454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4: No bootstrap required, opened a new log
I20260925 15:15:08.964097 8454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdcc30b5b404963bb6305e71c2535c4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36425 } }
I20260925 15:15:08.964288 8454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:08.964370 8454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bdcc30b5b404963bb6305e71c2535c4, State: Initialized, Role: FOLLOWER
I20260925 15:15:08.964533 8454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdcc30b5b404963bb6305e71c2535c4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36425 } }
I20260925 15:15:08.964624 8454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:08.964669 8454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:08.964740 8454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:08.965561 8454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdcc30b5b404963bb6305e71c2535c4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36425 } }
I20260925 15:15:08.965720 8454 leader_election.cc:302] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bdcc30b5b404963bb6305e71c2535c4; no voters:
I20260925 15:15:08.965958 8454 leader_election.cc:288] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:08.965986 8459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:08.966100 8459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [term 1 LEADER]: Becoming Leader. State: Replica: 2bdcc30b5b404963bb6305e71c2535c4, State: Running, Role: LEADER
I20260925 15:15:08.966195 8459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdcc30b5b404963bb6305e71c2535c4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36425 } }
I20260925 15:15:08.966372 8454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:08.966671 8459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bdcc30b5b404963bb6305e71c2535c4. Latest consensus state: current_term: 1 leader_uuid: "2bdcc30b5b404963bb6305e71c2535c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdcc30b5b404963bb6305e71c2535c4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36425 } } }
I20260925 15:15:08.966744 8459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:08.966897 8461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bdcc30b5b404963bb6305e71c2535c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bdcc30b5b404963bb6305e71c2535c4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36425 } } }
I20260925 15:15:08.966990 8461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:08.967137 8463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:08.967993 8463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:08.970306 8463 catalog_manager.cc:1347] Generated new cluster ID: 70e47863531e42e4950ad90e7e69ddd7
I20260925 15:15:08.970383 8463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:15:08 dist-test-slave-c5d6 krb5kdc[8373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:08.978649 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:08.961493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.8.43.190:36425)
0925 15:15:08.961799 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:08.961810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:08.962075 (+ 265us) server_negotiation.cc:407] Connection header received
0925 15:15:08.964072 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:08.964078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:08.964148 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:08.964293 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:08.966265 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:08.968529 (+ 2264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:08.969713 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:08.970358 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:08.970453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:08.974077 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:08.974081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:08.974082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:08.974093 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:08.976507 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:08.976964 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:08.976968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:08.976968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:08.977095 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:08.977382 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:08.977382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:08.977383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:08.977624 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:08.977801 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:08.978161 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 15:15:08.978356 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260925 15:15:08.994644 8463 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:08.995379 8463 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:09.000684 8463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bdcc30b5b404963bb6305e71c2535c4: Generated new TSK 0
I20260925 15:15:09.000901 8463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:09.036019 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36425
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:09.143034 8480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:09.143302 8480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:09.143379 8480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:09.144497 8480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:09.144563 8480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:09.144614 8480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:09.146060 8480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:09.146320 8480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:09.146561 8480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:09.152185 8485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:09.152207 8486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:09.152335 8488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:09.152949 8480 server_base.cc:1034] running on GCE node
I20260925 15:15:09.153239 8480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:09.153517 8480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:09.154696 8480 hybrid_clock.cc:648] HybridClock initialized: now 1790349309154684 us; error 28 us; skew 500 ppm
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:09.157254 8480 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:09.157661 8480 webserver.cc:460] Webserver started at http://127.8.43.129:40227/ using document root <none> and password file <none>
I20260925 15:15:09.157888 8480 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:09.157955 8480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:09.158103 8480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:09.159107 8480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "c1781506aae945bd866c1c60fa1d110f"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "42044e3df09ad6e607b1059ca7793411"
server_key_iv: "a4cc5cf7f6028b960d394126c88b7d12"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.159518 8480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "c1781506aae945bd866c1c60fa1d110f"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "42044e3df09ad6e607b1059ca7793411"
server_key_iv: "a4cc5cf7f6028b960d394126c88b7d12"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.160817 8480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:09.161643 8495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.161834 8480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:09.161942 8480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "c1781506aae945bd866c1c60fa1d110f"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "42044e3df09ad6e607b1059ca7793411"
server_key_iv: "a4cc5cf7f6028b960d394126c88b7d12"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.162029 8480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:09.209517 8480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:09.211671 8480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:09.211894 8480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:09.212150 8480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:09.212486 8480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:09.212519 8480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.212541 8480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:09.212574 8480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.218401 8480 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:41661
I20260925 15:15:09.218500 8608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:41661 every 8 connection(s)
I20260925 15:15:09.218806 8480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:09.222054 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 8480
I20260925 15:15:09.222136 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:09.222317 8366 external_mini_cluster.cc:1428] Setting key 682e6417dab0fccc2d9b2fb68d531e3b
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:09.229365 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:09.219676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:43955 (local address 127.8.43.190:36425)
0925 15:15:09.219754 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:09.219756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:09.219944 (+ 188us) server_negotiation.cc:407] Connection header received
0925 15:15:09.220416 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:09.220417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:09.220427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:09.220478 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:09.221443 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.221961 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.223314 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.223553 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.223641 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:09.226108 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:09.226109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:09.226110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:09.226118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:09.227897 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.228298 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.228299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.228299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.228376 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.228640 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.228641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.228642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.228792 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:09.228872 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:09.229134 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:15:09.229241 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:15:09.229370 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:09.219625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36425 (local address 127.8.43.129:43955)
0925 15:15:09.219812 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:09.219850 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:09.220314 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:09.220534 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:09.220537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:09.221171 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:09.221364 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:09.221368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.222090 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.222091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:09.223185 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:09.223188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.223652 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.223653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:09.223789 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:09.224408 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:09.224420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:09.226013 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:09.227984 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:09.227991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:09.227993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:09.228220 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:09.228446 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:09.228447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:09.228447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:09.228544 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:09.228865 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:09.228870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:09.228997 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 15:15:09.229084 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:15:09.229204 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260925 15:15:09.230079 8609 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36425
I20260925 15:15:09.230185 8609 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:09.230403 8609 heartbeater.cc:511] Master 127.8.43.190:36425 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
I20260925 15:15:09.231413 8418 ts_manager.cc:194] Registered new tserver with Master: c1781506aae945bd866c1c60fa1d110f (127.8.43.129:41661)
I20260925 15:15:09.232175 8418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:43955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:09.249511 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36425
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:09.332114 8614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:09.332309 8614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:09.332332 8614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:09.333460 8614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:09.333503 8614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:09.333529 8614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:09.334856 8614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:09.335026 8614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:09.335191 8614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:09.340186 8622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:09.340193 8620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:09.340261 8614 server_base.cc:1034] running on GCE node
W20260925 15:15:09.340421 8619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:09.340692 8614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:09.340971 8614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:09.342123 8614 hybrid_clock.cc:648] HybridClock initialized: now 1790349309342106 us; error 29 us; skew 500 ppm
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:09.344952 8614 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:09.345516 8614 webserver.cc:460] Webserver started at http://127.8.43.130:45169/ using document root <none> and password file <none>
I20260925 15:15:09.345710 8614 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:09.345752 8614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:09.345847 8614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:09.346850 8614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "02780d7a2cc34c0f8b2d9baea6f8f3c6"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "8f848ed719de451c99dded5d2868d3b4"
server_key_iv: "ffc43f69389c48527272bfcfa5cf4dba"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.347196 8614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "02780d7a2cc34c0f8b2d9baea6f8f3c6"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "8f848ed719de451c99dded5d2868d3b4"
server_key_iv: "ffc43f69389c48527272bfcfa5cf4dba"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.348567 8614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:09.349362 8629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.349581 8614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:15:09.349676 8614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "02780d7a2cc34c0f8b2d9baea6f8f3c6"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "8f848ed719de451c99dded5d2868d3b4"
server_key_iv: "ffc43f69389c48527272bfcfa5cf4dba"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.349763 8614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:09.360734 8614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:09.363159 8614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:09.363370 8614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:09.363629 8614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:09.363958 8614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:09.364017 8614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.364056 8614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:09.364091 8614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.370615 8614 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:41669
I20260925 15:15:09.370651 8742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:41669 every 8 connection(s)
I20260925 15:15:09.370996 8614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:09.374363 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 8614
I20260925 15:15:09.374487 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:09.374678 8366 external_mini_cluster.cc:1428] Setting key a5aea4fd33f46f36b3f7c7770242f99e
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:09.381515 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:09.371859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:44485 (local address 127.8.43.190:36425)
0925 15:15:09.371919 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:09.371921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:09.372135 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:15:09.372706 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:09.372708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:09.372719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:09.372776 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:09.373728 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.374259 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.375674 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.375885 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.375967 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:09.378442 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:09.378443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:09.378444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:09.378453 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:09.380144 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.380571 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.380573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.380573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.380632 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.380891 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.380892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.380893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.381003 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:09.381080 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:09.381344 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:15:09.381426 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:15:09.381700 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:09.371843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36425 (local address 127.8.43.130:44485)
0925 15:15:09.372012 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:09.372045 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:09.372598 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:09.372849 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:09.372855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:09.373429 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:09.373617 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:09.373622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.374355 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.374356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:09.375579 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:09.375581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.375976 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.375977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:09.376107 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:09.376671 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:09.376682 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:09.378278 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:09.380242 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:09.380249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:09.380251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:09.380479 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:09.380699 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:09.380700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:09.380701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:09.380809 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:09.381070 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:09.381077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:09.381211 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 15:15:09.381317 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:15:09.381464 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260925 15:15:09.382206 8743 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36425
I20260925 15:15:09.382309 8743 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:09.382529 8743 heartbeater.cc:511] Master 127.8.43.190:36425 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
I20260925 15:15:09.382884 8418 ts_manager.cc:194] Registered new tserver with Master: 02780d7a2cc34c0f8b2d9baea6f8f3c6 (127.8.43.130:41669)
I20260925 15:15:09.383401 8418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:44485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:09.399298 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36425
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:09.479532 8748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:09.479748 8748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:09.479787 8748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:09.480926 8748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:09.480988 8748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:09.481030 8748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:09.482430 8748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:09.482676 8748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:09.482888 8748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:09.487776 8754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:09.487788 8753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:09.488000 8748 server_base.cc:1034] running on GCE node
W20260925 15:15:09.487833 8756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:09.488262 8748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:09.488504 8748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:09.489688 8748 hybrid_clock.cc:648] HybridClock initialized: now 1790349309489693 us; error 45 us; skew 500 ppm
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:09.492211 8748 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:09.492611 8748 webserver.cc:460] Webserver started at http://127.8.43.131:40509/ using document root <none> and password file <none>
I20260925 15:15:09.492821 8748 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:09.492893 8748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:09.493032 8748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:09.494081 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "256c2cb04b1d46df9e6889c8f7f1e35d"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "9eb2d5aed7ac618185261b49e9ed415e"
server_key_iv: "f80c04be1d410401f3e986423b0dc886"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.494426 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "256c2cb04b1d46df9e6889c8f7f1e35d"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "9eb2d5aed7ac618185261b49e9ed415e"
server_key_iv: "f80c04be1d410401f3e986423b0dc886"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.495774 8748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:09.496538 8763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.496699 8748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:09.496793 8748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "256c2cb04b1d46df9e6889c8f7f1e35d"
format_stamp: "Formatted at 2026-09-25 15:15:09 on dist-test-slave-c5d6"
server_key: "9eb2d5aed7ac618185261b49e9ed415e"
server_key_iv: "f80c04be1d410401f3e986423b0dc886"
server_key_version: "encryptionkey@0"
I20260925 15:15:09.496868 8748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:09.505851 8748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:09.507855 8748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:09.508024 8748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:09.508253 8748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:09.508558 8748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:09.508610 8748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.508651 8748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:09.508677 8748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:09.514462 8748 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:35089
I20260925 15:15:09.514532 8876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:35089 every 8 connection(s)
I20260925 15:15:09.514868 8748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:09.523906 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 8748
I20260925 15:15:09.523989 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:09.524060 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:09.515644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:41685 (local address 127.8.43.190:36425)
0925 15:15:09.515703 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:09.515705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:09.515937 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:15:09.516429 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:09.516430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:09.516441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:09.516508 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:09.517456 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.517959 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.518930 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.519119 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.519176 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:09.521340 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:09.521341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:09.521341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:09.521347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:09.522746 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.523143 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.523144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.523144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.523193 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.523485 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.523486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.523486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.523588 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:09.523640 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:09.523874 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:15:09.523953 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:15:09.524149 8366 external_mini_cluster.cc:1428] Setting key b498ff84fd864babaf0c3163c3c76b74
I20260925 15:15:09.524271 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:09.515634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36425 (local address 127.8.43.131:41685)
0925 15:15:09.515791 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:09.515835 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:09.516321 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:09.516564 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:09.516568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:09.517169 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:09.517361 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:09.517366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.518046 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.518047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:09.518841 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:09.518843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.519215 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.519268 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:09.519407 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:09.519892 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:09.519901 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:09.521254 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:09.522833 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:09.522839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:09.522840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:09.523073 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:09.523301 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:09.523302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:09.523302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:09.523425 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:09.523646 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:09.523650 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:09.523763 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 15:15:09.523845 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:15:09.524006 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 15:15:09.524822 8877 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36425
I20260925 15:15:09.524946 8877 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:09.525108 8877 heartbeater.cc:511] Master 127.8.43.190:36425 requested a full tablet report, sending...
I20260925 15:15:09.525521 8417 ts_manager.cc:194] Registered new tserver with Master: 256c2cb04b1d46df9e6889c8f7f1e35d (127.8.43.131:35089)
I20260925 15:15:09.525906 8417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:41685
I20260925 15:15:09.536430 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:09.558717 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:09.551100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.8.43.129:41661)
0925 15:15:09.551330 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:09.551335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:09.551345 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:15:09.551372 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:09.551374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:09.551387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:09.551451 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:09.551976 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.552672 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.553429 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.553738 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.553829 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:09.556078 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:09.556085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:09.556085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:09.556092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:09.557666 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.557982 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.557984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.557984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.558039 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.558255 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.558256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.558256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.558410 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:09.558528 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:09.558582 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:15:09.558641 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
W20260925 15:15:09.559190 8523 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42572
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:09.581280 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:09.574195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.8.43.129:41661)
0925 15:15:09.574270 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:09.574272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:09.574399 (+ 127us) server_negotiation.cc:407] Connection header received
0925 15:15:09.574498 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:09.574499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:09.574508 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:09.574582 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:09.575121 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.575688 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.576440 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:09.576652 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:09.576726 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:09.579066 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:09.579068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:09.579068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:09.579074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:09.580346 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.580668 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.580669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.580670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.580716 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:09.580906 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:09.580907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:09.580907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:09.580992 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:09.581075 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:09.581162 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:15:09.581217 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:15:09.581863 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 8480
I20260925 15:15:09.586632 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 8614
I20260925 15:15:09.590731 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 8748
I20260925 15:15:09.594765 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 8386
2026-09-25T15:15:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2997 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8891](info): set up 2 sockets
Sep 25 15:15:09 dist-test-slave-c5d6 krb5kdc[8891](info): commencing operation
krb5kdc: starting...
W20260925 15:15:10.622673 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:10 dist-test-slave-c5d6 krb5kdc[8891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:10.734663 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44423
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:10.813575 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:10.813774 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:10.813796 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:10.814935 8904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:10.814975 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:10.814990 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:10.815002 8904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:10.815014 8904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:10.816496 8904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:10.816692 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:10.816862 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:10.821676 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:10.821681 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:10.821694 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:10.821945 8904 server_base.cc:1034] running on GCE node
I20260925 15:15:10.822234 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:10.822525 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:10.823688 8904 hybrid_clock.cc:648] HybridClock initialized: now 1790349310823674 us; error 29 us; skew 500 ppm
Sep 25 15:15:10 dist-test-slave-c5d6 krb5kdc[8891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349310, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:10.829900 8904 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:10.830331 8904 webserver.cc:460] Webserver started at http://127.8.43.190:42733/ using document root <none> and password file <none>
I20260925 15:15:10.830557 8904 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:10.830622 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:10.830780 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:10.831907 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "c39749c93bfb4af29ec68f16a93fe7ac"
format_stamp: "Formatted at 2026-09-25 15:15:10 on dist-test-slave-c5d6"
server_key: "c19aa31e226ec0eced4233241112f1e6"
server_key_iv: "55e418f901385d1ca5072f862239e151"
server_key_version: "encryptionkey@0"
I20260925 15:15:10.832278 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "c39749c93bfb4af29ec68f16a93fe7ac"
format_stamp: "Formatted at 2026-09-25 15:15:10 on dist-test-slave-c5d6"
server_key: "c19aa31e226ec0eced4233241112f1e6"
server_key_iv: "55e418f901385d1ca5072f862239e151"
server_key_version: "encryptionkey@0"
I20260925 15:15:10.833573 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:10.834292 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:10.834478 8904 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:10.834579 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "c39749c93bfb4af29ec68f16a93fe7ac"
format_stamp: "Formatted at 2026-09-25 15:15:10 on dist-test-slave-c5d6"
server_key: "c19aa31e226ec0eced4233241112f1e6"
server_key_iv: "55e418f901385d1ca5072f862239e151"
server_key_version: "encryptionkey@0"
I20260925 15:15:10.834649 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:10.851853 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:10.853717 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:10.853880 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:10.857913 8904 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44423
I20260925 15:15:10.857969 8971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44423 every 8 connection(s)
I20260925 15:15:10.858457 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:10.858982 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 8904
I20260925 15:15:10.859081 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:10.859272 8972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:10.859298 8366 external_mini_cluster.cc:1428] Setting key ebb089340844eac6c768190e3b38dbcc
I20260925 15:15:10.862651 8972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac: Bootstrap starting.
I20260925 15:15:10.863337 8972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:10.863637 8972 log.cc:824] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:10.864388 8972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac: No bootstrap required, opened a new log
Sep 25 15:15:10 dist-test-slave-c5d6 krb5kdc[8891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:10.866660 8972 raft_consensus.cc:348] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39749c93bfb4af29ec68f16a93fe7ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44423 } }
I20260925 15:15:10.866809 8972 raft_consensus.cc:374] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:10.866832 8972 raft_consensus.cc:729] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39749c93bfb4af29ec68f16a93fe7ac, State: Initialized, Role: FOLLOWER
I20260925 15:15:10.866982 8972 consensus_queue.cc:260] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39749c93bfb4af29ec68f16a93fe7ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44423 } }
I20260925 15:15:10.867087 8972 raft_consensus.cc:388] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:10.867122 8972 raft_consensus.cc:482] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:10.867177 8972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:10.867898 8972 raft_consensus.cc:504] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39749c93bfb4af29ec68f16a93fe7ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44423 } }
I20260925 15:15:10.868024 8972 leader_election.cc:302] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c39749c93bfb4af29ec68f16a93fe7ac; no voters:
I20260925 15:15:10.868222 8972 leader_election.cc:288] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:10.868343 8977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:10.868476 8972 sys_catalog.cc:564] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:10.868494 8977 raft_consensus.cc:686] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [term 1 LEADER]: Becoming Leader. State: Replica: c39749c93bfb4af29ec68f16a93fe7ac, State: Running, Role: LEADER
I20260925 15:15:10.870062 8977 consensus_queue.cc:237] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39749c93bfb4af29ec68f16a93fe7ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44423 } }
I20260925 15:15:10.870435 8978 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader c39749c93bfb4af29ec68f16a93fe7ac. Latest consensus state: current_term: 1 leader_uuid: "c39749c93bfb4af29ec68f16a93fe7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39749c93bfb4af29ec68f16a93fe7ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44423 } } }
I20260925 15:15:10.870522 8978 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:10.870685 8979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c39749c93bfb4af29ec68f16a93fe7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39749c93bfb4af29ec68f16a93fe7ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44423 } } }
I20260925 15:15:10.870797 8979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:10.870798 8991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:10.871519 8991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:10.871585 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:10.859841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.8.43.190:44423)
0925 15:15:10.860037 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:10.860046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:10.860085 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:15:10.860546 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:10.860555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:10.860631 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:10.860784 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:10.861359 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:10.862715 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:10.863572 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:10.864134 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:10.864246 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:10.867071 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:10.867076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:10.867076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:10.867087 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:10.869426 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:10.869847 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:10.869850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:10.869850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:10.869952 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:10.870180 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:10.870181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:10.870181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:10.870514 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:10.870677 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:10.871168 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 15:15:10.871350 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260925 15:15:10.872963 8991 catalog_manager.cc:1347] Generated new cluster ID: b3a4008cd98147599a32f08d34eb3408
I20260925 15:15:10.873003 8991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:10.883141 8991 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:10.883678 8991 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:10.890100 8991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c39749c93bfb4af29ec68f16a93fe7ac: Generated new TSK 0
I20260925 15:15:10.890306 8991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:10.918663 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44423
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:10.998955 8998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:10.999187 8998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:10.999228 8998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:11.000404 8998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:11.000471 8998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:11.000517 8998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:11.002007 8998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44423
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:11.002271 8998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:11.002502 8998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:11.007637 9006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:11.007645 9003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:11.007632 9004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:11.007999 8998 server_base.cc:1034] running on GCE node
I20260925 15:15:11.008226 8998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:11.008457 8998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:11.009598 8998 hybrid_clock.cc:648] HybridClock initialized: now 1790349311009589 us; error 26 us; skew 500 ppm
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:11.012043 8998 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:11.012490 8998 webserver.cc:460] Webserver started at http://127.8.43.129:35563/ using document root <none> and password file <none>
I20260925 15:15:11.012720 8998 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:11.012786 8998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:11.012941 8998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:11.013959 8998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "cb5c62b3c6d14cd8884b8c90f0d566df"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "21b0a43d27726c78a7d51d82f081237c"
server_key_iv: "cd4bc594e952051516a896d38c30c46e"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.014300 8998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "cb5c62b3c6d14cd8884b8c90f0d566df"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "21b0a43d27726c78a7d51d82f081237c"
server_key_iv: "cd4bc594e952051516a896d38c30c46e"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.015666 8998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 15:15:11.016413 9013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.016599 8998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:11.016698 8998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "cb5c62b3c6d14cd8884b8c90f0d566df"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "21b0a43d27726c78a7d51d82f081237c"
server_key_iv: "cd4bc594e952051516a896d38c30c46e"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.016768 8998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:11.032498 8998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:11.034423 8998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:11.034584 8998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:11.034811 8998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:11.035111 8998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:11.035163 8998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.035205 8998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:11.035285 8998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.041160 8998 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:42493
I20260925 15:15:11.041200 9126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:42493 every 8 connection(s)
I20260925 15:15:11.041520 8998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:11.043448 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 8998
I20260925 15:15:11.043560 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:11.043708 8366 external_mini_cluster.cc:1428] Setting key 0b9a8e170d5846528dff37a8daab0956
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
I20260925 15:15:11.051901 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.042232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:59599 (local address 127.8.43.190:44423)
0925 15:15:11.042298 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.042300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.042524 (+ 224us) server_negotiation.cc:407] Connection header received
0925 15:15:11.043576 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.043578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.043590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.043646 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:11.044428 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.044921 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.046366 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.046576 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.046634 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.049100 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:11.049102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:11.049102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:11.049110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:11.050508 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.050938 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.050939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.050940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.051000 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.051287 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.051289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.051289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.051439 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:11.051520 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.051753 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.051828 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:15:11.052008 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.042230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44423 (local address 127.8.43.129:59599)
0925 15:15:11.042391 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:11.042427 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:11.043464 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:11.043717 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:11.043722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:11.044076 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:11.044334 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.044340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.045020 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.045021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.046271 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.046273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.046660 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.046661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.046808 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.047369 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:11.047379 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:11.049003 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:11.050610 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:11.050616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:11.050618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:11.050852 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:11.051066 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:11.051067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:11.051068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:11.051193 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:11.051512 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:11.051517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:11.051628 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 15:15:11.051706 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:15:11.051817 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260925 15:15:11.052656 9127 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44423
I20260925 15:15:11.052737 9127 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:11.052928 9127 heartbeater.cc:511] Master 127.8.43.190:44423 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
I20260925 15:15:11.053493 8936 ts_manager.cc:194] Registered new tserver with Master: cb5c62b3c6d14cd8884b8c90f0d566df (127.8.43.129:42493)
I20260925 15:15:11.054392 8936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:59599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:11.067754 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44423
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:11.146102 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:11.146292 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:11.146315 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:11.147476 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:11.147514 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:11.147540 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:11.148983 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44423
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:11.149224 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:11.149451 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:11.154161 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:11.154165 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:11.154165 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:11.154484 9132 server_base.cc:1034] running on GCE node
I20260925 15:15:11.154701 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:11.154954 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:11.156144 9132 hybrid_clock.cc:648] HybridClock initialized: now 1790349311156130 us; error 24 us; skew 500 ppm
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:11.159020 9132 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:11.159574 9132 webserver.cc:460] Webserver started at http://127.8.43.130:36221/ using document root <none> and password file <none>
I20260925 15:15:11.159855 9132 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:11.159922 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:11.160070 9132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:11.161432 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "d21fc4d088c2439daf1f65676e17007f"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "07fecf90b1d3918cb5d38a35056494b6"
server_key_iv: "997dfa85e0a0bfe1b490d0ebd8462909"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.161760 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "d21fc4d088c2439daf1f65676e17007f"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "07fecf90b1d3918cb5d38a35056494b6"
server_key_iv: "997dfa85e0a0bfe1b490d0ebd8462909"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.163148 9132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:11.163928 9147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.164080 9132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:11.164167 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "d21fc4d088c2439daf1f65676e17007f"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "07fecf90b1d3918cb5d38a35056494b6"
server_key_iv: "997dfa85e0a0bfe1b490d0ebd8462909"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.164220 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:11.181010 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:11.183331 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:11.183446 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:11.183610 9132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:11.183899 9132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:11.183933 9132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.183981 9132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:11.184011 9132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.189848 9132 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:39119
I20260925 15:15:11.189898 9260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:39119 every 8 connection(s)
I20260925 15:15:11.190217 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:11.192436 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 9132
I20260925 15:15:11.192513 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:11.192679 8366 external_mini_cluster.cc:1428] Setting key 2dd4e5ba9bf9bba69ff9a01f2f4ebe9c
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
I20260925 15:15:11.201265 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.190913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:40741 (local address 127.8.43.190:44423)
0925 15:15:11.190957 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.190960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.191143 (+ 183us) server_negotiation.cc:407] Connection header received
0925 15:15:11.192178 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.192179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.192194 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.192245 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:11.193047 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.193681 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.195196 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.195558 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.195645 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.198213 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:11.198215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:11.198215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:11.198223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:11.199722 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.200185 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.200187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.200187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.200249 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.200523 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.200524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.200524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.200647 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:11.200723 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.201051 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.201152 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:15:11.201318 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.190871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44423 (local address 127.8.43.130:40741)
0925 15:15:11.191003 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:11.191053 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:11.192056 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:11.192317 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:11.192322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:11.192673 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:11.192947 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.192954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.193776 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.193777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.195106 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.195109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.195665 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.195666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.195785 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.196350 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:11.196368 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:11.198092 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:11.199830 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:11.199839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:11.199841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:11.200090 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:11.200320 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:11.200321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:11.200322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:11.200448 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:11.200713 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:11.200720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:11.200878 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 15:15:11.201001 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:15:11.201144 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260925 15:15:11.201756 9261 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44423
I20260925 15:15:11.201848 9261 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:11.202075 9261 heartbeater.cc:511] Master 127.8.43.190:44423 requested a full tablet report, sending...
I20260925 15:15:11.202484 8936 ts_manager.cc:194] Registered new tserver with Master: d21fc4d088c2439daf1f65676e17007f (127.8.43.130:39119)
I20260925 15:15:11.202940 8936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:40741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:11.217584 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44423
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:11.297369 9266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:11.297610 9266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:11.297647 9266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:11.298787 9266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:11.298868 9266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:11.298914 9266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:11.300434 9266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44423
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:11.300694 9266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:11.300932 9266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:11.305773 9271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:11.305797 9274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:11.305770 9272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:11.306152 9266 server_base.cc:1034] running on GCE node
I20260925 15:15:11.306365 9266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:11.306600 9266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:11.307758 9266 hybrid_clock.cc:648] HybridClock initialized: now 1790349311307739 us; error 28 us; skew 500 ppm
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:11.310235 9266 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:11.310724 9266 webserver.cc:460] Webserver started at http://127.8.43.131:35169/ using document root <none> and password file <none>
I20260925 15:15:11.310947 9266 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:11.311014 9266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:11.311168 9266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:11.312291 9266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "3b03ec023010495c836e34ba93f1c6e0"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "1ce2a1a40f1690fac51c56d68f8ab50e"
server_key_iv: "a2359f039a7fa9e0d69e6e7e0af4441d"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.312657 9266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "3b03ec023010495c836e34ba93f1c6e0"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "1ce2a1a40f1690fac51c56d68f8ab50e"
server_key_iv: "a2359f039a7fa9e0d69e6e7e0af4441d"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.314000 9266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:11.314752 9281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.314947 9266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:11.315035 9266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "3b03ec023010495c836e34ba93f1c6e0"
format_stamp: "Formatted at 2026-09-25 15:15:11 on dist-test-slave-c5d6"
server_key: "1ce2a1a40f1690fac51c56d68f8ab50e"
server_key_iv: "a2359f039a7fa9e0d69e6e7e0af4441d"
server_key_version: "encryptionkey@0"
I20260925 15:15:11.315088 9266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:11.338307 9266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:11.340282 9266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:11.340395 9266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:11.340574 9266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:11.340821 9266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:11.340852 9266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.340873 9266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:11.340885 9266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:11.346683 9266 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:41705
I20260925 15:15:11.346728 9394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:41705 every 8 connection(s)
I20260925 15:15:11.347038 9266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:11.352332 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 9266
I20260925 15:15:11.352504 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:11.352667 8366 external_mini_cluster.cc:1428] Setting key 36c88b8e253cbad0ef367cfca5a09f24
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:11.356204 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.347776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:60503 (local address 127.8.43.190:44423)
0925 15:15:11.347816 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.347818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.348029 (+ 211us) server_negotiation.cc:407] Connection header received
0925 15:15:11.348968 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.348969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.348979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.349038 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:11.349621 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.350153 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.351080 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.351309 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.351363 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.353658 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:11.353660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:11.353660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:11.353668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:11.354793 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.355237 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.355238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.355238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.355296 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.355573 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.355574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.355574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.355690 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:11.355764 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.356035 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.356111 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:15:11.356396 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.347753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44423 (local address 127.8.43.131:60503)
0925 15:15:11.347907 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:11.347947 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:11.348846 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:11.349111 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:11.349114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:11.349362 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:11.349551 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.349555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.350227 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.350227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.351006 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.351008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.351379 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.351380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.351481 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.351992 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:11.351999 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:11.353542 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:11.354891 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:11.354896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:11.354899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:11.355114 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:11.355390 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:11.355391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:11.355392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:11.355504 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:11.355755 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:11.355761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:11.355902 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:15:11.356010 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:15:11.356160 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260925 15:15:11.356791 9395 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44423
I20260925 15:15:11.356882 9395 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:11.357082 9395 heartbeater.cc:511] Master 127.8.43.190:44423 requested a full tablet report, sending...
I20260925 15:15:11.357440 8936 ts_manager.cc:194] Registered new tserver with Master: 3b03ec023010495c836e34ba93f1c6e0 (127.8.43.131:41705)
I20260925 15:15:11.357805 8936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:60503
I20260925 15:15:11.364586 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:15:11.372979 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.366436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.8.43.190:44423)
0925 15:15:11.366510 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.366513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.366585 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:15:11.366666 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.366667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.366676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.366751 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:11.367401 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.367960 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.368777 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.369051 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.369124 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.370413 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:11.370414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:11.370415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:11.370421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:11.371764 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.372107 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.372108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.372108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.372162 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.372400 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.372401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.372401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.372548 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:11.372669 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.372789 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.372876 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:15:11.375605 8936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:11.377254 8936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:11.384177 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.379957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.8.43.130:39119)
0925 15:15:11.380111 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.380116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.380440 (+ 324us) server_negotiation.cc:407] Connection header received
0925 15:15:11.380545 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.380548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.380565 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.380639 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:11.380806 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.381590 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.382955 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.383704 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.383748 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.383799 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.384049 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.384091 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260925 15:15:11.384225 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.380046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41705 (local address 127.0.0.1:51526)
0925 15:15:11.380558 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:11.380566 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:11.380610 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:11.380822 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:11.380823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:11.380824 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:11.380858 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.380860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.382110 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.382111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.383154 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.383156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.383888 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.383888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.383949 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.383968 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 15:15:11.384027 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:15:11.384088 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":35,"threads_started":1}
I20260925 15:15:11.384471 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.380129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.8.43.131:41705)
0925 15:15:11.380293 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.380298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.380617 (+ 319us) server_negotiation.cc:407] Connection header received
0925 15:15:11.380670 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.380672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.380688 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.380798 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:11.380921 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.382004 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.383251 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.383788 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.383888 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.383944 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.384348 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.384395 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260925 15:15:11.384491 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.379818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:42493 (local address 127.0.0.1:50834)
0925 15:15:11.380318 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:11.380381 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:11.380457 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:11.380931 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:11.380932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:11.380934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:11.380982 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.380983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.382210 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.382211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.383308 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.383311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.384262 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.384264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.384352 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.384355 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:15:11.384400 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 15:15:11.384431 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"spinlock_wait_cycles":768,"thread_start_us":68,"threads_started":1}
I20260925 15:15:11.384531 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.379818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:39119 (local address 127.0.0.1:54254)
0925 15:15:11.380322 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:11.380359 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:11.380453 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:11.380673 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:11.380675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:11.380677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:11.380747 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.380748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.381705 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.381706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.382854 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.382856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.383871 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.383872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.383948 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.383960 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:15:11.384032 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:15:11.384103 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":27,"thread_start_us":67,"threads_started":1}
I20260925 15:15:11.385231 9329 tablet_service.cc:1467] Processing CreateTablet for tablet 7db52cf0a1cb43b4aa6c04edafb009a9 (DEFAULT_TABLE table=test-table [id=a4d601d9743a4586a46395ab1e870954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:11.385233 9195 tablet_service.cc:1467] Processing CreateTablet for tablet 7db52cf0a1cb43b4aa6c04edafb009a9 (DEFAULT_TABLE table=test-table [id=a4d601d9743a4586a46395ab1e870954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:11.385571 9329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db52cf0a1cb43b4aa6c04edafb009a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:11.385571 9195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db52cf0a1cb43b4aa6c04edafb009a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:11.387391 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.380150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.8.43.129:42493)
0925 15:15:11.380295 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.380300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.380452 (+ 152us) server_negotiation.cc:407] Connection header received
0925 15:15:11.380749 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.380752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.380766 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.380848 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:11.381063 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.382118 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.383403 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.384114 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.384176 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.384228 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.384799 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.387318 (+ 2519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260925 15:15:11.387954 9061 tablet_service.cc:1467] Processing CreateTablet for tablet 7db52cf0a1cb43b4aa6c04edafb009a9 (DEFAULT_TABLE table=test-table [id=a4d601d9743a4586a46395ab1e870954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:11.388250 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db52cf0a1cb43b4aa6c04edafb009a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:11.389031 9410 tablet_bootstrap.cc:492] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f: Bootstrap starting.
I20260925 15:15:11.389683 9410 tablet_bootstrap.cc:654] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:11.390403 9410 log.cc:824] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:11.388811 9409 tablet_bootstrap.cc:492] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0: Bootstrap starting.
I20260925 15:15:11.391302 9409 tablet_bootstrap.cc:654] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:11.391429 9410 tablet_bootstrap.cc:492] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f: No bootstrap required, opened a new log
I20260925 15:15:11.391543 9410 ts_tablet_manager.cc:1397] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:15:11.391762 9412 tablet_bootstrap.cc:492] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df: Bootstrap starting.
I20260925 15:15:11.391976 9409 log.cc:824] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:11.392416 9412 tablet_bootstrap.cc:654] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:11.392683 9409 tablet_bootstrap.cc:492] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0: No bootstrap required, opened a new log
I20260925 15:15:11.392757 9409 ts_tablet_manager.cc:1397] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 15:15:11.392971 9412 log.cc:824] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:11.393720 9412 tablet_bootstrap.cc:492] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df: No bootstrap required, opened a new log
I20260925 15:15:11.393784 9412 ts_tablet_manager.cc:1397] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:11.393687 9410 raft_consensus.cc:348] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.393870 9410 raft_consensus.cc:374] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:11.393918 9410 raft_consensus.cc:729] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21fc4d088c2439daf1f65676e17007f, State: Initialized, Role: FOLLOWER
I20260925 15:15:11.394055 9410 consensus_queue.cc:260] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.394330 9410 ts_tablet_manager.cc:1428] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:11.394457 9261 heartbeater.cc:503] Master 127.8.43.190:44423 was elected leader, sending a full tablet report...
I20260925 15:15:11.394591 9409 raft_consensus.cc:348] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.394773 9409 raft_consensus.cc:374] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:11.394814 9409 raft_consensus.cc:729] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b03ec023010495c836e34ba93f1c6e0, State: Initialized, Role: FOLLOWER
I20260925 15:15:11.394932 9409 consensus_queue.cc:260] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.395191 9409 ts_tablet_manager.cc:1428] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 15:15:11.395196 9395 heartbeater.cc:503] Master 127.8.43.190:44423 was elected leader, sending a full tablet report...
I20260925 15:15:11.395972 9412 raft_consensus.cc:348] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.396128 9412 raft_consensus.cc:374] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:11.396180 9412 raft_consensus.cc:729] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5c62b3c6d14cd8884b8c90f0d566df, State: Initialized, Role: FOLLOWER
I20260925 15:15:11.396284 9412 consensus_queue.cc:260] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.396550 9127 heartbeater.cc:503] Master 127.8.43.190:44423 was elected leader, sending a full tablet report...
I20260925 15:15:11.396559 9412 ts_tablet_manager.cc:1428] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 15:15:11.440743 9262 tablet.cc:2369] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:11.515908 9417 raft_consensus.cc:482] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:11.516069 9417 raft_consensus.cc:504] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.516573 9417 leader_election.cc:288] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b03ec023010495c836e34ba93f1c6e0 (127.8.43.131:41705), d21fc4d088c2439daf1f65676e17007f (127.8.43.130:39119)
I20260925 15:15:11.519712 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.516920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:37789 (local address 127.8.43.130:39119)
0925 15:15:11.516964 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.516967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.516975 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:11.517039 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.517040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.517062 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.517120 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:11.517298 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.517770 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.518825 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.519380 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.519437 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.519444 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.519601 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.519644 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:15:11.519730 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.516760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:39119 (local address 127.8.43.129:37789)
0925 15:15:11.516906 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:11.516917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:11.516968 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:11.517132 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:11.517133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:11.517137 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:11.517228 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.517230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.517864 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.517865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.518760 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.518762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.519454 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.519456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.519527 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.519531 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:11.519588 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:15:11.519617 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":15,"thread_start_us":67,"threads_started":1}
I20260925 15:15:11.519775 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.516871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:47527 (local address 127.8.43.131:41705)
0925 15:15:11.516914 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.516917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.516929 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:15:11.516963 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.516964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.516976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.517031 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:11.517223 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.517797 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.518960 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.519429 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.519501 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.519510 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.519665 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.519702 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:15:11.520114 9215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db52cf0a1cb43b4aa6c04edafb009a9" candidate_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d21fc4d088c2439daf1f65676e17007f" is_pre_election: true
I20260925 15:15:11.520172 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.516760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41705 (local address 127.8.43.129:47527)
0925 15:15:11.516821 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:11.516835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:11.516902 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:11.517055 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:11.517056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:11.517058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:11.517140 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.517143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.517908 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.517909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.518877 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:11.518879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.519512 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.519512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:11.519589 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.519593 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:11.519648 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:15:11.519681 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":53}
I20260925 15:15:11.520283 9215 raft_consensus.cc:2454] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb5c62b3c6d14cd8884b8c90f0d566df in term 0.
I20260925 15:15:11.520476 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db52cf0a1cb43b4aa6c04edafb009a9" candidate_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b03ec023010495c836e34ba93f1c6e0" is_pre_election: true
I20260925 15:15:11.520609 9017 leader_election.cc:302] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb5c62b3c6d14cd8884b8c90f0d566df, d21fc4d088c2439daf1f65676e17007f; no voters:
I20260925 15:15:11.520668 9349 raft_consensus.cc:2454] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb5c62b3c6d14cd8884b8c90f0d566df in term 0.
I20260925 15:15:11.520740 9417 raft_consensus.cc:2781] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:11.520792 9417 raft_consensus.cc:482] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:11.520826 9417 raft_consensus.cc:3037] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:11.521498 9417 raft_consensus.cc:504] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.521642 9417 leader_election.cc:288] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [CANDIDATE]: Term 1 election: Requested vote from peers 3b03ec023010495c836e34ba93f1c6e0 (127.8.43.131:41705), d21fc4d088c2439daf1f65676e17007f (127.8.43.130:39119)
I20260925 15:15:11.521771 9215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db52cf0a1cb43b4aa6c04edafb009a9" candidate_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d21fc4d088c2439daf1f65676e17007f"
I20260925 15:15:11.521787 9349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db52cf0a1cb43b4aa6c04edafb009a9" candidate_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b03ec023010495c836e34ba93f1c6e0"
I20260925 15:15:11.521865 9215 raft_consensus.cc:3037] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:11.521886 9349 raft_consensus.cc:3037] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:11.522584 9349 raft_consensus.cc:2454] T 7db52cf0a1cb43b4aa6c04edafb009a9 P 3b03ec023010495c836e34ba93f1c6e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb5c62b3c6d14cd8884b8c90f0d566df in term 1.
I20260925 15:15:11.522598 9215 raft_consensus.cc:2454] T 7db52cf0a1cb43b4aa6c04edafb009a9 P d21fc4d088c2439daf1f65676e17007f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb5c62b3c6d14cd8884b8c90f0d566df in term 1.
I20260925 15:15:11.522770 9017 leader_election.cc:302] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb5c62b3c6d14cd8884b8c90f0d566df, d21fc4d088c2439daf1f65676e17007f; no voters:
I20260925 15:15:11.522861 9417 raft_consensus.cc:2781] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:11.523053 9417 raft_consensus.cc:686] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [term 1 LEADER]: Becoming Leader. State: Replica: cb5c62b3c6d14cd8884b8c90f0d566df, State: Running, Role: LEADER
I20260925 15:15:11.523129 9417 consensus_queue.cc:237] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } }
I20260925 15:15:11.524039 8935 catalog_manager.cc:5495] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df reported cstate change: term changed from 0 to 1, leader changed from <none> to cb5c62b3c6d14cd8884b8c90f0d566df (127.8.43.129). New cstate: current_term: 1 leader_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb5c62b3c6d14cd8884b8c90f0d566df" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b03ec023010495c836e34ba93f1c6e0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d21fc4d088c2439daf1f65676e17007f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39119 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:11.535199 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.527745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.8.43.129:42493)
0925 15:15:11.527848 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.527851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.527961 (+ 110us) server_negotiation.cc:407] Connection header received
0925 15:15:11.528031 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.528031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.528043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.528128 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:11.528654 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.529118 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.529881 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.530182 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.530271 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.532692 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:11.532698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:11.532698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:11.532704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:11.534274 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.534601 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.534604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.534604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.534655 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.534800 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.534800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.534801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.534909 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:11.535022 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.535053 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.535106 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 15:15:11.542167 9128 tablet.cc:2369] T 7db52cf0a1cb43b4aa6c04edafb009a9 P cb5c62b3c6d14cd8884b8c90f0d566df: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:11.557767 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.550878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.8.43.129:42493)
0925 15:15:11.550946 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.550948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.551058 (+ 110us) server_negotiation.cc:407] Connection header received
0925 15:15:11.551109 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.551109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.551120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.551176 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:11.551854 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.552317 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.553065 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.553238 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.553291 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.555455 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:11.555456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:11.555456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:11.555462 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:11.556770 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.557114 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.557115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.557115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.557162 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.557332 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.557333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.557333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.557434 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:11.557504 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.557650 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.557706 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[8891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:11.579773 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:11.573192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.8.43.129:42493)
0925 15:15:11.573249 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:11.573250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:11.573290 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:15:11.573384 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:11.573385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:11.573394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:11.573440 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:11.573928 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.574417 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.575109 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:11.575394 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:11.575452 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:11.577484 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:11.577485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:11.577485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:11.577491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:11.578813 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.579093 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.579094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.579094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.579142 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:11.579359 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:11.579359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:11.579360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:11.579444 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:11.579514 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:11.579602 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:15:11.579666 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 15:15:11.580324 9041 tablet_service.cc:2908] Rejecting scan request for tablet 7db52cf0a1cb43b4aa6c04edafb009a9: Uninitialized: clean time has not yet been initialized
I20260925 15:15:11.581352 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 8998
I20260925 15:15:11.586593 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 9132
I20260925 15:15:11.590567 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 9266
I20260925 15:15:11.594872 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 8904
2026-09-25T15:15:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2000 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[9433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[9433](info): set up 2 sockets
Sep 25 15:15:11 dist-test-slave-c5d6 krb5kdc[9433](info): commencing operation
krb5kdc: starting...
W20260925 15:15:12.658990 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:12 dist-test-slave-c5d6 krb5kdc[9433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:12.770025 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:43797
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:12.849753 9446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:12.849992 9446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:12.850026 9446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:12.851306 9446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:12.851363 9446 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:15:12.851384 9446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:12.851406 9446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:12.851436 9446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:12.851467 9446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:12.852912 9446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:43797
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:12.853169 9446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:12.853387 9446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:12.858140 9452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:12.858152 9451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:12.858119 9454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:12.858407 9446 server_base.cc:1034] running on GCE node
I20260925 15:15:12.858634 9446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:12.858917 9446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:12.860059 9446 hybrid_clock.cc:648] HybridClock initialized: now 1790349312860030 us; error 48 us; skew 500 ppm
Sep 25 15:15:12 dist-test-slave-c5d6 krb5kdc[9433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349312, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:12.862417 9446 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:12.862877 9446 webserver.cc:460] Webserver started at http://127.8.43.190:33459/ using document root <none> and password file <none>
I20260925 15:15:12.863087 9446 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:12.863152 9446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:12.863313 9446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:12.864338 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "deafd5bab0d24c99b5283b711fb2978c"
format_stamp: "Formatted at 2026-09-25 15:15:12 on dist-test-slave-c5d6"
server_key: "2023219a118a27a4a055f23454058571"
server_key_iv: "f382323c19a741f4e923732153d312a6"
server_key_version: "encryptionkey@0"
I20260925 15:15:12.864683 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "deafd5bab0d24c99b5283b711fb2978c"
format_stamp: "Formatted at 2026-09-25 15:15:12 on dist-test-slave-c5d6"
server_key: "2023219a118a27a4a055f23454058571"
server_key_iv: "f382323c19a741f4e923732153d312a6"
server_key_version: "encryptionkey@0"
I20260925 15:15:12.865993 9446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:12.866669 9461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:12.866830 9446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:12.866931 9446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "deafd5bab0d24c99b5283b711fb2978c"
format_stamp: "Formatted at 2026-09-25 15:15:12 on dist-test-slave-c5d6"
server_key: "2023219a118a27a4a055f23454058571"
server_key_iv: "f382323c19a741f4e923732153d312a6"
server_key_version: "encryptionkey@0"
I20260925 15:15:12.867010 9446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:12.882354 9446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:12.884471 9446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:12.884645 9446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:12.889331 9446 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:43797
I20260925 15:15:12.889391 9523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:43797 every 8 connection(s)
I20260925 15:15:12.889752 9446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:12.890444 9524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:12.892685 9524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c: Bootstrap starting.
I20260925 15:15:12.893272 9524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:12.893567 9524 log.cc:824] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:12.894209 9524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c: No bootstrap required, opened a new log
I20260925 15:15:12.894536 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 9446
I20260925 15:15:12.894606 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:12.894773 8366 external_mini_cluster.cc:1428] Setting key 0a090bb03ba00d8e8a7fd81e7e2faf5b
I20260925 15:15:12.897008 9524 raft_consensus.cc:348] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deafd5bab0d24c99b5283b711fb2978c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43797 } }
I20260925 15:15:12.897202 9524 raft_consensus.cc:374] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:12.897239 9524 raft_consensus.cc:729] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deafd5bab0d24c99b5283b711fb2978c, State: Initialized, Role: FOLLOWER
I20260925 15:15:12.897367 9524 consensus_queue.cc:260] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deafd5bab0d24c99b5283b711fb2978c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43797 } }
I20260925 15:15:12.897449 9524 raft_consensus.cc:388] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:12.897471 9524 raft_consensus.cc:482] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:12.897537 9524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:12.898108 9524 raft_consensus.cc:504] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deafd5bab0d24c99b5283b711fb2978c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43797 } }
I20260925 15:15:12.898209 9524 leader_election.cc:302] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deafd5bab0d24c99b5283b711fb2978c; no voters:
I20260925 15:15:12.898372 9524 leader_election.cc:288] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:12.898447 9529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:12.898583 9529 raft_consensus.cc:686] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [term 1 LEADER]: Becoming Leader. State: Replica: deafd5bab0d24c99b5283b711fb2978c, State: Running, Role: LEADER
I20260925 15:15:12.898619 9524 sys_catalog.cc:564] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:12.898667 9529 consensus_queue.cc:237] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deafd5bab0d24c99b5283b711fb2978c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43797 } }
I20260925 15:15:12.899087 9530 sys_catalog.cc:455] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deafd5bab0d24c99b5283b711fb2978c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deafd5bab0d24c99b5283b711fb2978c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43797 } } }
I20260925 15:15:12.899204 9530 sys_catalog.cc:458] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:12.899649 9536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:12.899087 9531 sys_catalog.cc:455] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [sys.catalog]: SysCatalogTable state changed. Reason: New leader deafd5bab0d24c99b5283b711fb2978c. Latest consensus state: current_term: 1 leader_uuid: "deafd5bab0d24c99b5283b711fb2978c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deafd5bab0d24c99b5283b711fb2978c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43797 } } }
I20260925 15:15:12.900061 9531 sys_catalog.cc:458] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:12.900135 9536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:15:12 dist-test-slave-c5d6 krb5kdc[9433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:12.904726 9536 catalog_manager.cc:1347] Generated new cluster ID: 7471515e3c9a4d03ba0ae6d0bfea6252
I20260925 15:15:12.904779 9536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:12.906783 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:12.895285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.8.43.190:43797)
0925 15:15:12.895469 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:12.895480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:12.895517 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:15:12.896972 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:12.896981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:12.897047 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:12.897219 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:12.897998 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:12.899451 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:12.900518 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:12.901036 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:12.901159 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:12.903470 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:12.903473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:12.903474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:12.903485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:12.905295 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:12.905646 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:12.905649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:12.905650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:12.905751 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:12.905961 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:12.905962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:12.905962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:12.906199 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:12.906367 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:12.906455 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:15:12.906601 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260925 15:15:12.912490 9536 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:12.913190 9536 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:12.921977 9536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c: Generated new TSK 0
I20260925 15:15:12.922173 9536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:12.949792 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:43797
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:13.031136 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:13.031397 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:13.031437 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:13.032598 9550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:13.032667 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:13.032704 9550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:15:13.032758 9550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:13.034247 9550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:13.034531 9550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:13.034762 9550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:13.039744 9558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:13.039744 9556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:13.039855 9555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:13.040019 9550 server_base.cc:1034] running on GCE node
I20260925 15:15:13.040208 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:13.040407 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:13.041553 9550 hybrid_clock.cc:648] HybridClock initialized: now 1790349313041540 us; error 29 us; skew 500 ppm
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:13.044059 9550 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:13.044561 9550 webserver.cc:460] Webserver started at http://127.8.43.129:40853/ using document root <none> and password file <none>
I20260925 15:15:13.044795 9550 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:13.044862 9550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:13.045001 9550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:13.046041 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "1d323df359eb4dac9fa2454d33223def"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "be0b0437c25de3ae29725edca73dec29"
server_key_iv: "6b8d1d6d8ecfaa7ad4883c12f84f6e10"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.046404 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "1d323df359eb4dac9fa2454d33223def"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "be0b0437c25de3ae29725edca73dec29"
server_key_iv: "6b8d1d6d8ecfaa7ad4883c12f84f6e10"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.047845 9550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:13.048542 9565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.048702 9550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:13.048797 9550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "1d323df359eb4dac9fa2454d33223def"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "be0b0437c25de3ae29725edca73dec29"
server_key_iv: "6b8d1d6d8ecfaa7ad4883c12f84f6e10"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.048877 9550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:13.074621 9550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:13.076602 9550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:13.076768 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:13.077304 9550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:13.077366 9550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.077410 9550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:13.077464 9550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.084410 9550 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:44537
I20260925 15:15:13.084518 9680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:44537 every 8 connection(s)
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:13.084791 9550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:13.085031 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 9550
I20260925 15:15:13.085119 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:13.085282 8366 external_mini_cluster.cc:1428] Setting key 94212e1de877c984035874f68d17c603
I20260925 15:15:13.088541 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.077838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:60337 (local address 127.8.43.190:43797)
0925 15:15:13.077908 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.077911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.078126 (+ 215us) server_negotiation.cc:407] Connection header received
0925 15:15:13.078644 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.078646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.078662 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.078740 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.079645 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.080484 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.082160 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.082491 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.082548 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.085404 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.085406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.085406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.085415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.087054 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.087579 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.087580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.087581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.087650 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.087889 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.087890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.087890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.088047 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.088114 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.088359 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.088466 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:15:13.088717 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.077811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:43797 (local address 127.8.43.129:60337)
0925 15:15:13.077995 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.078034 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.078544 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.078843 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.078847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.079200 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:13.079551 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.079557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.080595 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.080596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.082054 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.082058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.082577 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.082578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.082743 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.083392 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:13.083404 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:13.085297 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:13.087127 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.087134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.087136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.087500 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.087719 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.087720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.087721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.087818 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.088107 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:13.088111 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:13.088219 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.088293 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.088438 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
I20260925 15:15:13.089265 9681 heartbeater.cc:347] Connected to a master server at 127.8.43.190:43797
I20260925 15:15:13.089355 9681 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:13.089543 9681 heartbeater.cc:511] Master 127.8.43.190:43797 requested a full tablet report, sending...
I20260925 15:15:13.090054 9478 ts_manager.cc:194] Registered new tserver with Master: 1d323df359eb4dac9fa2454d33223def (127.8.43.129:44537)
I20260925 15:15:13.090656 9478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:60337
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
I20260925 15:15:13.096947 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.091471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:43797 (local address 127.8.43.129:48839)
0925 15:15:13.091583 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.091592 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.091644 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.091821 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.091822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.091968 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:13.092012 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.092013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.092971 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.092972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.093460 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.093461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.093952 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.093953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.094006 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.094426 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:13.094431 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:13.094678 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:13.096055 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.096056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.096056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.096239 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.096421 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.096421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.096421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.096464 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.096675 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:13.096675 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:13.096740 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.096807 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.096877 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260925 15:15:13.097018 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.091660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:48839 (local address 127.8.43.190:43797)
0925 15:15:13.091703 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.091705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.091714 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:13.091735 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.091735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.091745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.091819 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.092080 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.092854 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.093543 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.093867 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.093942 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.094756 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.094757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.094757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.094763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.095965 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.096302 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.096303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.096303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.096348 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.096497 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.096497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.096497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.096617 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.096689 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.096836 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.096920 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:13.110319 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:43797
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:13.191615 9690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:13.191867 9690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:13.191908 9690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:13.193099 9690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:13.193174 9690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:13.193207 9690 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:15:13.193246 9690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:13.194660 9690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:13.194942 9690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:13.195181 9690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:13.200613 9696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:13.200632 9698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:13.200582 9695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:13.200769 9690 server_base.cc:1034] running on GCE node
I20260925 15:15:13.201002 9690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:13.201210 9690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:13.202389 9690 hybrid_clock.cc:648] HybridClock initialized: now 1790349313202369 us; error 50 us; skew 500 ppm
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:13.204912 9690 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:13.205365 9690 webserver.cc:460] Webserver started at http://127.8.43.130:46015/ using document root <none> and password file <none>
I20260925 15:15:13.205605 9690 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:13.205677 9690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:13.205849 9690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:13.206936 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "032747bfd70a4c9ebb1d687200ba52af"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "71043c44f97baa2142c936c5b6cda5c0"
server_key_iv: "eec4d39a0f8c9473ed44b7acad1e3e40"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.207381 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "032747bfd70a4c9ebb1d687200ba52af"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "71043c44f97baa2142c936c5b6cda5c0"
server_key_iv: "eec4d39a0f8c9473ed44b7acad1e3e40"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.208752 9690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:13.209511 9705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.209673 9690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.209769 9690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "032747bfd70a4c9ebb1d687200ba52af"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "71043c44f97baa2142c936c5b6cda5c0"
server_key_iv: "eec4d39a0f8c9473ed44b7acad1e3e40"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.209856 9690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:13.232105 9690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:13.234081 9690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:13.234241 9690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:13.234786 9690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:13.234886 9690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.234931 9690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:13.234959 9690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:13.242125 9690 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:33599
I20260925 15:15:13.242233 9820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:33599 every 8 connection(s)
I20260925 15:15:13.242545 9690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:13.245015 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.235344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:47423 (local address 127.8.43.190:43797)
0925 15:15:13.235413 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.235415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.235627 (+ 212us) server_negotiation.cc:407] Connection header received
0925 15:15:13.236134 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.236136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.236151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.236231 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.236891 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.237743 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.238776 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.239120 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.239196 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.241995 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.241997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.241997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.242006 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.243616 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.244030 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.244031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.244032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.244088 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.244354 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.244355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.244355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.244477 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.244551 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.244835 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.244923 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:15:13.245081 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 9690
I20260925 15:15:13.245179 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:13.245232 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.235307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:43797 (local address 127.8.43.130:47423)
0925 15:15:13.235482 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.235527 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.236028 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.236304 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.236308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.236624 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:13.236810 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.236815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.237876 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.237877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.238708 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.238710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.239255 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.239256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.239390 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.239944 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:13.239954 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:13.241884 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:13.243715 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.243721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.243723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.243949 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.244174 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.244175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.244175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.244281 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.244557 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:13.244564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:13.244699 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.244821 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.244981 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260925 15:15:13.245433 8366 external_mini_cluster.cc:1428] Setting key 5b2e166ed351800b68e31cef9ce78fea
I20260925 15:15:13.245882 9821 heartbeater.cc:347] Connected to a master server at 127.8.43.190:43797
I20260925 15:15:13.245986 9821 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:13.246157 9821 heartbeater.cc:511] Master 127.8.43.190:43797 requested a full tablet report, sending...
I20260925 15:15:13.246564 9477 ts_manager.cc:194] Registered new tserver with Master: 032747bfd70a4c9ebb1d687200ba52af (127.8.43.130:33599)
I20260925 15:15:13.246999 9477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:47423
I20260925 15:15:13.254041 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.247416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:43797 (local address 127.8.43.130:60093)
0925 15:15:13.247965 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.247976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.248045 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.248164 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.248165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.248389 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:13.248455 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.248457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.249088 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.249089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.249634 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.249636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.250166 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.250167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.250217 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.250656 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:13.250660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:13.250925 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:13.252983 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.252984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.252984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.253146 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.253404 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.253404 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.253404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.253457 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.253810 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:13.253811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:13.253846 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.253898 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.253970 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":64,"threads_started":1}
I20260925 15:15:13.254110 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.247469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:60093 (local address 127.8.43.190:43797)
0925 15:15:13.247508 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.247510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.248050 (+ 540us) server_negotiation.cc:407] Connection header received
0925 15:15:13.248098 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.248099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.248109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.248150 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.248533 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.248995 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.249728 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.250056 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.250135 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.251016 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.251017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.251018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.251025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.252895 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.253226 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.253227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.253227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.253297 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.253526 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.253527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.253528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.253736 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.253824 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.253929 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.254005 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:13.275076 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:43797
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:13.358139 9830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:13.358349 9830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:13.358376 9830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:13.359561 9830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:13.359602 9830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:13.359638 9830 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:15:13.359658 9830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:13.360996 9830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:43797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:13.361176 9830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:13.361356 9830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:13.366382 9838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:13.366549 9830 server_base.cc:1034] running on GCE node
W20260925 15:15:13.366391 9835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:13.366602 9836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:13.366899 9830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:13.367183 9830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:13.368433 9830 hybrid_clock.cc:648] HybridClock initialized: now 1790349313368373 us; error 47 us; skew 500 ppm
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:13.371330 9830 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:13.371810 9830 webserver.cc:460] Webserver started at http://127.8.43.131:36699/ using document root <none> and password file <none>
I20260925 15:15:13.372031 9830 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:13.372074 9830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:13.372206 9830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:13.373250 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "c32c1442908643e9986a970ab45d1b21"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "310f8b06e3eeac3689182af6f2bd8c76"
server_key_iv: "4819085d8ccaf8bb004fc380a7d50aaf"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.373608 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "c32c1442908643e9986a970ab45d1b21"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "310f8b06e3eeac3689182af6f2bd8c76"
server_key_iv: "4819085d8ccaf8bb004fc380a7d50aaf"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.374925 9830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:15:13.375741 9845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.375931 9830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.376029 9830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "c32c1442908643e9986a970ab45d1b21"
format_stamp: "Formatted at 2026-09-25 15:15:13 on dist-test-slave-c5d6"
server_key: "310f8b06e3eeac3689182af6f2bd8c76"
server_key_iv: "4819085d8ccaf8bb004fc380a7d50aaf"
server_key_version: "encryptionkey@0"
I20260925 15:15:13.376111 9830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:13.392863 9830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:13.395360 9830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:13.395483 9830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:13.395936 9830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:13.396001 9830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.396046 9830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:13.396072 9830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.403425 9830 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:41023
I20260925 15:15:13.403476 9960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:41023 every 8 connection(s)
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:13.403805 9830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:13.406908 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.396507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:43797 (local address 127.8.43.131:47441)
0925 15:15:13.396695 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.396745 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.397301 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.397619 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.397624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.397986 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:13.398192 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.398197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.399335 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.399336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.400884 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.400887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.401456 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.401457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.401617 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.402267 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:13.402278 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:13.404280 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:13.405640 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.405646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.405648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.405860 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.406033 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.406033 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.406034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.406127 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.406434 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:13.406438 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:13.406547 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.406628 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.406744 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260925 15:15:13.406924 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.396493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:47441 (local address 127.8.43.190:43797)
0925 15:15:13.396536 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.396538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.396852 (+ 314us) server_negotiation.cc:407] Connection header received
0925 15:15:13.397430 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.397432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.397447 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.397530 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.398290 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.399190 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.400973 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.401295 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.401450 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.404376 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.404377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.404378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.404384 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.405551 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.405927 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.405927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.405928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.405977 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.406205 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.406206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.406206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.406371 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.406434 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.406687 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.406812 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:13.407450 9961 heartbeater.cc:347] Connected to a master server at 127.8.43.190:43797
I20260925 15:15:13.407536 9961 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:13.407719 9961 heartbeater.cc:511] Master 127.8.43.190:43797 requested a full tablet report, sending...
I20260925 15:15:13.408154 9477 ts_manager.cc:194] Registered new tserver with Master: c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023)
I20260925 15:15:13.408643 9477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:47441
I20260925 15:15:13.410162 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 9830
I20260925 15:15:13.410228 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:13.410441 8366 external_mini_cluster.cc:1428] Setting key 1b25a12cc9c4861ca33200dcd897a65c
I20260925 15:15:13.411266 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:15:13.414340 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.408556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:33721 (local address 127.8.43.190:43797)
0925 15:15:13.408598 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.408600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.408673 (+ 73us) server_negotiation.cc:407] Connection header received
0925 15:15:13.408758 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.408760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.408770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.408829 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.409215 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.409660 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.410361 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.410656 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.410730 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.411707 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.411708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.411709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.411717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.413282 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.413695 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.413696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.413696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.413742 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.413929 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.413930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.413930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.414020 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.414069 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.414229 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.414283 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:15:13.414440 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.408496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:43797 (local address 127.8.43.131:33721)
0925 15:15:13.408619 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.408629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.408689 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.408858 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.408859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.409078 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:13.409142 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.409144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.409743 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.409744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.410285 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.410287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.410747 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.410748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.410826 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.411304 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:13.411309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:13.411623 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:13.413358 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.413359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.413359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.413625 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.413802 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.413803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.413804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.413873 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.414081 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:13.414082 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:13.414132 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.414206 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.414313 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:13.433815 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.427320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.8.43.190:43797)
0925 15:15:13.427388 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.427390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.427398 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:13.427428 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.427429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.427437 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.427503 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.428039 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.428519 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.429255 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.429440 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.429544 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.431686 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.431687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.431688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.431694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.432819 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.433121 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.433121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.433122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.433171 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.433331 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.433331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.433331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.433429 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.433502 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.433693 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.433753 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:15:13.435412 9477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:13.436973 9477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:13.443746 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.439396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:33599 (local address 127.0.0.1:43830)
0925 15:15:13.439646 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.439683 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.439778 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.440039 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.440042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.440044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:13.440118 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.440120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.441493 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.441494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.442491 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.442494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.443412 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.443413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.443500 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.443513 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.443586 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.443649 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":24,"thread_start_us":57,"threads_started":1}
I20260925 15:15:13.443745 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.439407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44537 (local address 127.0.0.1:50464)
0925 15:15:13.439832 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.439842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.439925 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.440188 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.440189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.440191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:13.440249 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.440251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.441195 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.441195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.442491 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.442497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.443428 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.443428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.443502 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.443513 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.443586 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.443649 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":4,"thread_start_us":94,"threads_started":1}
I20260925 15:15:13.443770 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.439646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.8.43.129:44537)
0925 15:15:13.439813 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.439817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.439918 (+ 101us) server_negotiation.cc:407] Connection header received
0925 15:15:13.440013 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.440015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.440035 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.440117 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:13.440354 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.441108 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.442603 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.443267 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.443317 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.443360 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.443614 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.443658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260925 15:15:13.444063 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.439407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41023 (local address 127.0.0.1:55322)
0925 15:15:13.440027 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.440036 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.440084 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.440353 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.440354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.440356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:13.440398 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.440399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.441607 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.441609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.442785 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.442787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.443813 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.443815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.443888 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.443893 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.443950 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.443992 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":9,"thread_start_us":80,"threads_started":1}
I20260925 15:15:13.444201 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.439483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.8.43.130:33599)
0925 15:15:13.439621 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.439626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.439869 (+ 243us) server_negotiation.cc:407] Connection header received
0925 15:15:13.439924 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.439926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.439941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.440037 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:13.440257 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.441402 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.442608 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.443329 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.443403 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.443459 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.444065 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.444109 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260925 15:15:13.444293 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.439633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.8.43.131:41023)
0925 15:15:13.439896 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.439899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.440149 (+ 250us) server_negotiation.cc:407] Connection header received
0925 15:15:13.440193 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.440196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.440211 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.440287 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:13.440481 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.441520 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.442865 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.443660 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.443727 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.443779 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.444158 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.444202 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260925 15:15:13.444700 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 77485f2aea2443b4aa08cf38f6305683 (DEFAULT_TABLE table=test-table [id=9af8d3edb33849978e05fbc5b979b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:13.444831 9755 tablet_service.cc:1467] Processing CreateTablet for tablet 77485f2aea2443b4aa08cf38f6305683 (DEFAULT_TABLE table=test-table [id=9af8d3edb33849978e05fbc5b979b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:13.445062 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77485f2aea2443b4aa08cf38f6305683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.445180 9755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77485f2aea2443b4aa08cf38f6305683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.447906 9895 tablet_service.cc:1467] Processing CreateTablet for tablet 77485f2aea2443b4aa08cf38f6305683 (DEFAULT_TABLE table=test-table [id=9af8d3edb33849978e05fbc5b979b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:13.448233 9895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77485f2aea2443b4aa08cf38f6305683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.448292 9984 tablet_bootstrap.cc:492] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Bootstrap starting.
I20260925 15:15:13.448909 9984 tablet_bootstrap.cc:654] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.449230 9984 log.cc:824] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:13.449977 9984 tablet_bootstrap.cc:492] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: No bootstrap required, opened a new log
I20260925 15:15:13.450047 9984 ts_tablet_manager.cc:1397] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:13.451462 9985 tablet_bootstrap.cc:492] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: Bootstrap starting.
I20260925 15:15:13.451450 9987 tablet_bootstrap.cc:492] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: Bootstrap starting.
I20260925 15:15:13.452051 9984 raft_consensus.cc:348] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.452216 9984 raft_consensus.cc:374] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.452193 9985 tablet_bootstrap.cc:654] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.452270 9984 raft_consensus.cc:729] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032747bfd70a4c9ebb1d687200ba52af, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.452406 9984 consensus_queue.cc:260] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.452482 9987 tablet_bootstrap.cc:654] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.452666 9984 ts_tablet_manager.cc:1428] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:13.452665 9985 log.cc:824] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:13.452795 9821 heartbeater.cc:503] Master 127.8.43.190:43797 was elected leader, sending a full tablet report...
I20260925 15:15:13.452792 9987 log.cc:824] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:13.453536 9985 tablet_bootstrap.cc:492] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: No bootstrap required, opened a new log
I20260925 15:15:13.453605 9985 ts_tablet_manager.cc:1397] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:13.453891 9987 tablet_bootstrap.cc:492] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: No bootstrap required, opened a new log
I20260925 15:15:13.453958 9987 ts_tablet_manager.cc:1397] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:15:13.455163 9985 raft_consensus.cc:348] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.455348 9985 raft_consensus.cc:374] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.455374 9985 raft_consensus.cc:729] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d323df359eb4dac9fa2454d33223def, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.455332 9987 raft_consensus.cc:348] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.455510 9987 raft_consensus.cc:374] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.455499 9985 consensus_queue.cc:260] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.455545 9987 raft_consensus.cc:729] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32c1442908643e9986a970ab45d1b21, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.455653 9987 consensus_queue.cc:260] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.455752 9985 ts_tablet_manager.cc:1428] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:13.455906 9987 ts_tablet_manager.cc:1428] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:13.455896 9681 heartbeater.cc:503] Master 127.8.43.190:43797 was elected leader, sending a full tablet report...
I20260925 15:15:13.456054 9961 heartbeater.cc:503] Master 127.8.43.190:43797 was elected leader, sending a full tablet report...
W20260925 15:15:13.493364 9822 tablet.cc:2369] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:13.547991 9988 raft_consensus.cc:482] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:13.548115 9988 raft_consensus.cc:504] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.548564 9988 leader_election.cc:288] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023), 1d323df359eb4dac9fa2454d33223def (127.8.43.129:44537)
I20260925 15:15:13.551643 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.548873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:48625 (local address 127.8.43.129:44537)
0925 15:15:13.548939 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.548941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.549126 (+ 185us) server_negotiation.cc:407] Connection header received
0925 15:15:13.549154 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.549155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.549168 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.549240 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:13.549418 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.549907 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.550828 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.551210 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.551340 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.551350 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.551536 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.551577 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:15:13.551643 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.548812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:40551 (local address 127.8.43.131:41023)
0925 15:15:13.548965 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.548967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.548979 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:15:13.549010 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.549011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.549024 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.549085 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:13.549324 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.549832 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.550800 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.551323 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.551389 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.551399 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.551536 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.551575 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 15:15:13.551654 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.548779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44537 (local address 127.8.43.130:48625)
0925 15:15:13.548968 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.548979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.549046 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.549248 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.549248 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.549250 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:13.549336 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.549338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.549982 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.549983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.550716 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.550718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.551332 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.551333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.551439 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.551444 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.551503 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.551544 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":119,"threads_started":1}
I20260925 15:15:13.551656 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.548747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41023 (local address 127.8.43.130:40551)
0925 15:15:13.548804 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.548815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.548873 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.549152 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.549153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.549155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:13.549238 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.549240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.549925 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.549926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.550705 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.550707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.551395 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.551396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.551465 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.551468 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.551522 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.551547 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260925 15:15:13.552069 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77485f2aea2443b4aa08cf38f6305683" candidate_uuid: "032747bfd70a4c9ebb1d687200ba52af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d323df359eb4dac9fa2454d33223def" is_pre_election: true
I20260925 15:15:13.552217 9635 raft_consensus.cc:2454] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032747bfd70a4c9ebb1d687200ba52af in term 0.
I20260925 15:15:13.552414 9915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77485f2aea2443b4aa08cf38f6305683" candidate_uuid: "032747bfd70a4c9ebb1d687200ba52af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c32c1442908643e9986a970ab45d1b21" is_pre_election: true
I20260925 15:15:13.552438 9706 leader_election.cc:302] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032747bfd70a4c9ebb1d687200ba52af, 1d323df359eb4dac9fa2454d33223def; no voters:
I20260925 15:15:13.552558 9915 raft_consensus.cc:2454] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032747bfd70a4c9ebb1d687200ba52af in term 0.
I20260925 15:15:13.552592 9988 raft_consensus.cc:2781] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:13.552645 9988 raft_consensus.cc:482] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:13.552665 9988 raft_consensus.cc:3037] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.553740 9988 raft_consensus.cc:504] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.553881 9988 leader_election.cc:288] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [CANDIDATE]: Term 1 election: Requested vote from peers c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023), 1d323df359eb4dac9fa2454d33223def (127.8.43.129:44537)
I20260925 15:15:13.554005 9915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77485f2aea2443b4aa08cf38f6305683" candidate_uuid: "032747bfd70a4c9ebb1d687200ba52af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c32c1442908643e9986a970ab45d1b21"
I20260925 15:15:13.554054 9635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77485f2aea2443b4aa08cf38f6305683" candidate_uuid: "032747bfd70a4c9ebb1d687200ba52af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d323df359eb4dac9fa2454d33223def"
I20260925 15:15:13.554101 9915 raft_consensus.cc:3037] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.554116 9635 raft_consensus.cc:3037] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.554805 9915 raft_consensus.cc:2454] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032747bfd70a4c9ebb1d687200ba52af in term 1.
I20260925 15:15:13.554872 9635 raft_consensus.cc:2454] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032747bfd70a4c9ebb1d687200ba52af in term 1.
I20260925 15:15:13.555024 9706 leader_election.cc:302] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032747bfd70a4c9ebb1d687200ba52af, 1d323df359eb4dac9fa2454d33223def; no voters:
I20260925 15:15:13.555177 9988 raft_consensus.cc:2781] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:13.555394 9988 raft_consensus.cc:686] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 1 LEADER]: Becoming Leader. State: Replica: 032747bfd70a4c9ebb1d687200ba52af, State: Running, Role: LEADER
I20260925 15:15:13.555473 9988 consensus_queue.cc:237] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } }
I20260925 15:15:13.556402 9478 catalog_manager.cc:5495] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af reported cstate change: term changed from 0 to 1, leader changed from <none> to 032747bfd70a4c9ebb1d687200ba52af (127.8.43.130). New cstate: current_term: 1 leader_uuid: "032747bfd70a4c9ebb1d687200ba52af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } health_report { overall_health: UNKNOWN } } }
I20260925 15:15:13.563055 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.560669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.8.43.130:33599)
0925 15:15:13.560730 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.560731 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.560740 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:13.560799 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.560800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.560826 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.560890 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:13.561015 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.561471 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.562208 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.562409 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.562470 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.562677 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:13.562808 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:13.562897 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.562961 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.562983 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:15:13.565357 9635 raft_consensus.cc:1261] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Refusing update from remote peer 032747bfd70a4c9ebb1d687200ba52af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:13.565380 9915 raft_consensus.cc:1261] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Refusing update from remote peer 032747bfd70a4c9ebb1d687200ba52af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:13.565727 9994 consensus_queue.cc:1035] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:13.565842 9988 consensus_queue.cc:1035] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:13.571842 9478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:13.571949 9478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:13.572827 9478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c: Sending DeleteTablet for 3 replicas of tablet 77485f2aea2443b4aa08cf38f6305683
I20260925 15:15:13.573197 9755 tablet_service.cc:1514] Processing DeleteTablet for tablet 77485f2aea2443b4aa08cf38f6305683 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:13 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:43830
I20260925 15:15:13.573203 9895 tablet_service.cc:1514] Processing DeleteTablet for tablet 77485f2aea2443b4aa08cf38f6305683 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:13 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:55322
I20260925 15:15:13.573454 10006 tablet_replica.cc:330] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: stopping tablet replica
I20260925 15:15:13.573446 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 77485f2aea2443b4aa08cf38f6305683 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:13 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:50464
I20260925 15:15:13.573587 10006 raft_consensus.cc:2229] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:13.573681 10006 raft_consensus.cc:2258] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:13.573774 10008 tablet_replica.cc:330] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: stopping tablet replica
I20260925 15:15:13.573817 10007 tablet_replica.cc:330] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: stopping tablet replica
I20260925 15:15:13.573869 10008 raft_consensus.cc:2229] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:13.573916 10007 raft_consensus.cc:2229] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:13.573957 10008 raft_consensus.cc:2258] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:13.574023 10007 raft_consensus.cc:2258] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:13.574043 9478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:13.574180 9478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:13.574430 10008 ts_tablet_manager.cc:1905] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:13.574451 10007 ts_tablet_manager.cc:1905] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:13.575873 9755 tablet_service.cc:1467] Processing CreateTablet for tablet 27098dfe365f416782799fc02e328186 (DEFAULT_TABLE table=test-table [id=fde508b93df046688adc832feb5b8582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:13.575990 9755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27098dfe365f416782799fc02e328186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.576045 9895 tablet_service.cc:1467] Processing CreateTablet for tablet 27098dfe365f416782799fc02e328186 (DEFAULT_TABLE table=test-table [id=fde508b93df046688adc832feb5b8582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:13.576134 9895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27098dfe365f416782799fc02e328186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.576509 10007 ts_tablet_manager.cc:1918] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:13.576566 10007 log.cc:1196] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/77485f2aea2443b4aa08cf38f6305683
I20260925 15:15:13.576843 10007 ts_tablet_manager.cc:1939] T 77485f2aea2443b4aa08cf38f6305683 P c32c1442908643e9986a970ab45d1b21: Deleting consensus metadata
I20260925 15:15:13.577358 9987 tablet_bootstrap.cc:492] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: Bootstrap starting.
I20260925 15:15:13.577659 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 27098dfe365f416782799fc02e328186 (DEFAULT_TABLE table=test-table [id=fde508b93df046688adc832feb5b8582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:13.577759 9987 tablet_bootstrap.cc:654] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.577770 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27098dfe365f416782799fc02e328186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.578073 9464 catalog_manager.cc:4841] TS c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023): tablet 77485f2aea2443b4aa08cf38f6305683 (table test-table [id=9af8d3edb33849978e05fbc5b979b342]) successfully deleted
I20260925 15:15:13.578433 9987 tablet_bootstrap.cc:492] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: No bootstrap required, opened a new log
I20260925 15:15:13.578495 9987 ts_tablet_manager.cc:1397] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:13.578646 9987 raft_consensus.cc:348] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.578722 9987 raft_consensus.cc:374] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.578748 9987 raft_consensus.cc:729] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32c1442908643e9986a970ab45d1b21, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.578787 9987 consensus_queue.cc:260] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.578873 9987 ts_tablet_manager.cc:1428] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:13.578966 9984 tablet_bootstrap.cc:492] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: Bootstrap starting.
I20260925 15:15:13.579372 9984 tablet_bootstrap.cc:654] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.579772 9985 tablet_bootstrap.cc:492] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: Bootstrap starting.
I20260925 15:15:13.579907 10006 ts_tablet_manager.cc:1905] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:13.579957 9984 tablet_bootstrap.cc:492] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: No bootstrap required, opened a new log
I20260925 15:15:13.579988 9984 ts_tablet_manager.cc:1397] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:13.580121 9984 raft_consensus.cc:348] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.580153 9985 tablet_bootstrap.cc:654] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.580183 9984 raft_consensus.cc:374] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.580205 9984 raft_consensus.cc:729] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032747bfd70a4c9ebb1d687200ba52af, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.580248 9984 consensus_queue.cc:260] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.580325 9984 ts_tablet_manager.cc:1428] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.581037 10008 ts_tablet_manager.cc:1918] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:13.581099 10008 log.cc:1196] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/77485f2aea2443b4aa08cf38f6305683
I20260925 15:15:13.581341 10008 ts_tablet_manager.cc:1939] T 77485f2aea2443b4aa08cf38f6305683 P 1d323df359eb4dac9fa2454d33223def: Deleting consensus metadata
I20260925 15:15:13.581699 10006 ts_tablet_manager.cc:1918] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:13.581694 9462 catalog_manager.cc:4841] TS 1d323df359eb4dac9fa2454d33223def (127.8.43.129:44537): tablet 77485f2aea2443b4aa08cf38f6305683 (table test-table [id=9af8d3edb33849978e05fbc5b979b342]) successfully deleted
I20260925 15:15:13.581781 10006 log.cc:1196] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/77485f2aea2443b4aa08cf38f6305683
I20260925 15:15:13.582005 10006 ts_tablet_manager.cc:1939] T 77485f2aea2443b4aa08cf38f6305683 P 032747bfd70a4c9ebb1d687200ba52af: Deleting consensus metadata
I20260925 15:15:13.582109 9985 tablet_bootstrap.cc:492] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: No bootstrap required, opened a new log
I20260925 15:15:13.582163 9985 ts_tablet_manager.cc:1397] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:13.582296 9985 raft_consensus.cc:348] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.582367 9985 raft_consensus.cc:374] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.582389 9985 raft_consensus.cc:729] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d323df359eb4dac9fa2454d33223def, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.582429 9985 consensus_queue.cc:260] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.582516 9985 ts_tablet_manager.cc:1428] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.582647 9465 catalog_manager.cc:4841] TS 032747bfd70a4c9ebb1d687200ba52af (127.8.43.130:33599): tablet 77485f2aea2443b4aa08cf38f6305683 (table test-table [id=9af8d3edb33849978e05fbc5b979b342]) successfully deleted
W20260925 15:15:13.585603 9682 tablet.cc:2369] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:13.594568 9991 raft_consensus.cc:482] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:13.594645 9991 raft_consensus.cc:504] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.594967 9991 leader_election.cc:288] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032747bfd70a4c9ebb1d687200ba52af (127.8.43.130:33599), c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023)
I20260925 15:15:13.598884 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.595304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:57895 (local address 127.8.43.130:33599)
0925 15:15:13.595418 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.595420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.595649 (+ 229us) server_negotiation.cc:407] Connection header received
0925 15:15:13.595752 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.595753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.595776 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.595827 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:13.595999 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.596471 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.597973 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.598564 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.598642 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.598651 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.598778 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.598818 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:15:13.598965 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.595139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:33599 (local address 127.8.43.129:57895)
0925 15:15:13.595602 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.595610 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.595672 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.595888 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.595889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.595890 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:13.595929 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.595930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.596554 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.596555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.597883 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.597886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.598640 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.598641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.598696 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.598700 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.598773 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.598814 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":56,"thread_start_us":56,"threads_started":1}
I20260925 15:15:13.599428 9775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27098dfe365f416782799fc02e328186" candidate_uuid: "1d323df359eb4dac9fa2454d33223def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032747bfd70a4c9ebb1d687200ba52af" is_pre_election: true
I20260925 15:15:13.599584 9775 raft_consensus.cc:2454] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d323df359eb4dac9fa2454d33223def in term 0.
I20260925 15:15:13.599745 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.595372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:59707 (local address 127.8.43.131:41023)
0925 15:15:13.595439 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.595441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.595499 (+ 58us) server_negotiation.cc:407] Connection header received
0925 15:15:13.595578 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.595579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.595587 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.595661 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:13.595829 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.596587 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.598690 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.599306 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.599401 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.599408 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.599636 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.599677 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:15:13.599774 9569 leader_election.cc:302] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032747bfd70a4c9ebb1d687200ba52af, 1d323df359eb4dac9fa2454d33223def; no voters:
I20260925 15:15:13.599896 9991 raft_consensus.cc:2781] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:13.599934 9991 raft_consensus.cc:482] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:13.599985 9991 raft_consensus.cc:3037] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.600132 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.595139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41023 (local address 127.8.43.129:59707)
0925 15:15:13.595439 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.595451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.595517 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.595665 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.595666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.595668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:13.595747 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.595748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.596664 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.596665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.597883 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.597886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.599474 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.599475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.599552 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.599556 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.599877 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.599922 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":4,"thread_start_us":60,"threads_started":1}
I20260925 15:15:13.600342 9915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27098dfe365f416782799fc02e328186" candidate_uuid: "1d323df359eb4dac9fa2454d33223def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c32c1442908643e9986a970ab45d1b21" is_pre_election: true
I20260925 15:15:13.600430 9915 raft_consensus.cc:2454] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d323df359eb4dac9fa2454d33223def in term 0.
I20260925 15:15:13.600888 9991 raft_consensus.cc:504] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.601038 9991 leader_election.cc:288] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [CANDIDATE]: Term 1 election: Requested vote from peers 032747bfd70a4c9ebb1d687200ba52af (127.8.43.130:33599), c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023)
I20260925 15:15:13.601194 9775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27098dfe365f416782799fc02e328186" candidate_uuid: "1d323df359eb4dac9fa2454d33223def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032747bfd70a4c9ebb1d687200ba52af"
I20260925 15:15:13.601207 9915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27098dfe365f416782799fc02e328186" candidate_uuid: "1d323df359eb4dac9fa2454d33223def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c32c1442908643e9986a970ab45d1b21"
I20260925 15:15:13.601261 9775 raft_consensus.cc:3037] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.601269 9915 raft_consensus.cc:3037] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.602031 9915 raft_consensus.cc:2454] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d323df359eb4dac9fa2454d33223def in term 1.
I20260925 15:15:13.602020 9775 raft_consensus.cc:2454] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d323df359eb4dac9fa2454d33223def in term 1.
I20260925 15:15:13.602211 9569 leader_election.cc:302] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032747bfd70a4c9ebb1d687200ba52af, 1d323df359eb4dac9fa2454d33223def; no voters:
I20260925 15:15:13.602293 9991 raft_consensus.cc:2781] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:13.602392 9991 raft_consensus.cc:686] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 1 LEADER]: Becoming Leader. State: Replica: 1d323df359eb4dac9fa2454d33223def, State: Running, Role: LEADER
I20260925 15:15:13.602463 9991 consensus_queue.cc:237] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.603111 9477 catalog_manager.cc:5495] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d323df359eb4dac9fa2454d33223def (127.8.43.129). New cstate: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349312, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:13.631450 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.624052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.8.43.190:43797)
0925 15:15:13.624121 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.624124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.624211 (+ 87us) server_negotiation.cc:407] Connection header received
0925 15:15:13.624278 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.624279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.624291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.624347 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.624816 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.625602 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.626451 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.626752 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.626825 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.629023 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.629025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.629025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.629031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.630317 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.630752 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.630753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.630753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.630799 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.630994 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.630994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.630994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.631092 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.631141 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.631318 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.631387 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:15:13.631578 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.623962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:43797 (local address 127.0.0.1:52294)
0925 15:15:13.624132 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.624143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.624209 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.624363 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.624364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.624696 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:13.624741 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.624743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.625699 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.625700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.626375 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.626377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.626831 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.626832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.626883 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.627473 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:13.627478 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:13.628934 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:13.630396 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.630403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.630405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.630678 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.630868 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:13.630868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:13.630869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:13.630937 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:13.631149 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:13.631156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:13.631205 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.631301 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.631473 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":60,"thread_start_us":112,"threads_started":1}
I20260925 15:15:13.633672 9477 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:52294:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:15:13.633904 9477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:13.635308 9755 tablet_service.cc:1467] Processing CreateTablet for tablet 2b210b1f697c4580bf9d62814b2baf65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35719715c67f41969cae639d2a50f3d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:15:13.635334 9615 tablet_service.cc:1467] Processing CreateTablet for tablet 2b210b1f697c4580bf9d62814b2baf65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35719715c67f41969cae639d2a50f3d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:15:13.635483 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b210b1f697c4580bf9d62814b2baf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.635483 9755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b210b1f697c4580bf9d62814b2baf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.636622 9984 tablet_bootstrap.cc:492] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: Bootstrap starting.
I20260925 15:15:13.636713 9895 tablet_service.cc:1467] Processing CreateTablet for tablet 2b210b1f697c4580bf9d62814b2baf65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35719715c67f41969cae639d2a50f3d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:15:13.636878 9895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b210b1f697c4580bf9d62814b2baf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:13.636934 9985 tablet_bootstrap.cc:492] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: Bootstrap starting.
I20260925 15:15:13.637032 9984 tablet_bootstrap.cc:654] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.637336 9985 tablet_bootstrap.cc:654] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.638300 9987 tablet_bootstrap.cc:492] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: Bootstrap starting.
I20260925 15:15:13.638703 9987 tablet_bootstrap.cc:654] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:13.638731 9984 tablet_bootstrap.cc:492] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: No bootstrap required, opened a new log
I20260925 15:15:13.638796 9984 ts_tablet_manager.cc:1397] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:13.638940 9984 raft_consensus.cc:348] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.639030 9984 raft_consensus.cc:374] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.639057 9984 raft_consensus.cc:729] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032747bfd70a4c9ebb1d687200ba52af, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.639104 9984 consensus_queue.cc:260] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.639210 9984 ts_tablet_manager.cc:1428] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:13.639329 9994 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.639437 9994 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:13.639473 9987 tablet_bootstrap.cc:492] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: No bootstrap required, opened a new log
I20260925 15:15:13.639515 9987 ts_tablet_manager.cc:1397] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:13.639662 9987 raft_consensus.cc:348] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.639730 9987 raft_consensus.cc:374] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.639753 9987 raft_consensus.cc:729] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32c1442908643e9986a970ab45d1b21, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.639794 9987 consensus_queue.cc:260] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.639811 9985 tablet_bootstrap.cc:492] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: No bootstrap required, opened a new log
I20260925 15:15:13.639859 9985 ts_tablet_manager.cc:1397] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:15:13.639883 9987 ts_tablet_manager.cc:1428] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.640015 9985 raft_consensus.cc:348] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.640094 9985 raft_consensus.cc:374] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:13.640113 9985 raft_consensus.cc:729] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d323df359eb4dac9fa2454d33223def, State: Initialized, Role: FOLLOWER
I20260925 15:15:13.640154 9985 consensus_queue.cc:260] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.640242 9985 ts_tablet_manager.cc:1428] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:13.640413 9992 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.640520 9992 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:13.640653 9991 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.640728 9991 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 15:15:13.654460 9962 tablet.cc:2369] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:13.681941 9991 raft_consensus.cc:482] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:13.682052 9991 raft_consensus.cc:504] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.682202 9991 leader_election.cc:288] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032747bfd70a4c9ebb1d687200ba52af (127.8.43.130:33599), c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023)
I20260925 15:15:13.682451 9915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b210b1f697c4580bf9d62814b2baf65" candidate_uuid: "1d323df359eb4dac9fa2454d33223def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c32c1442908643e9986a970ab45d1b21" is_pre_election: true
I20260925 15:15:13.682442 9775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b210b1f697c4580bf9d62814b2baf65" candidate_uuid: "1d323df359eb4dac9fa2454d33223def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032747bfd70a4c9ebb1d687200ba52af" is_pre_election: true
I20260925 15:15:13.682550 9915 raft_consensus.cc:2454] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d323df359eb4dac9fa2454d33223def in term 0.
I20260925 15:15:13.682551 9775 raft_consensus.cc:2454] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d323df359eb4dac9fa2454d33223def in term 0.
I20260925 15:15:13.682711 9568 leader_election.cc:302] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d323df359eb4dac9fa2454d33223def, c32c1442908643e9986a970ab45d1b21; no voters:
I20260925 15:15:13.682821 9991 raft_consensus.cc:2781] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:13.682880 9991 raft_consensus.cc:482] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:13.682900 9991 raft_consensus.cc:3037] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.683739 9991 raft_consensus.cc:504] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.683869 9991 leader_election.cc:288] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [CANDIDATE]: Term 1 election: Requested vote from peers 032747bfd70a4c9ebb1d687200ba52af (127.8.43.130:33599), c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023)
I20260925 15:15:13.684062 9775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b210b1f697c4580bf9d62814b2baf65" candidate_uuid: "1d323df359eb4dac9fa2454d33223def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032747bfd70a4c9ebb1d687200ba52af"
I20260925 15:15:13.684067 9915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b210b1f697c4580bf9d62814b2baf65" candidate_uuid: "1d323df359eb4dac9fa2454d33223def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c32c1442908643e9986a970ab45d1b21"
I20260925 15:15:13.684135 9775 raft_consensus.cc:3037] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.684157 9915 raft_consensus.cc:3037] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:13.684757 9915 raft_consensus.cc:2454] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d323df359eb4dac9fa2454d33223def in term 1.
I20260925 15:15:13.684839 9775 raft_consensus.cc:2454] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d323df359eb4dac9fa2454d33223def in term 1.
I20260925 15:15:13.684926 9568 leader_election.cc:302] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d323df359eb4dac9fa2454d33223def, c32c1442908643e9986a970ab45d1b21; no voters:
I20260925 15:15:13.685029 9991 raft_consensus.cc:2781] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:13.685081 9991 raft_consensus.cc:686] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [term 1 LEADER]: Becoming Leader. State: Replica: 1d323df359eb4dac9fa2454d33223def, State: Running, Role: LEADER
I20260925 15:15:13.685130 9991 consensus_queue.cc:237] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } }
I20260925 15:15:13.685393 9991 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: TxnStatusTablet state changed. Reason: New leader 1d323df359eb4dac9fa2454d33223def. Latest consensus state: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.685456 9991 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:15:13.685652 10026 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:15:13.685782 10026 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:15:13.685819 9477 catalog_manager.cc:5495] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d323df359eb4dac9fa2454d33223def (127.8.43.129). New cstate: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } health_report { overall_health: UNKNOWN } } }
I20260925 15:15:13.702345 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.699931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.8.43.129:44537)
0925 15:15:13.700093 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.700095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.700100 (+ 5us) server_negotiation.cc:407] Connection header received
0925 15:15:13.700127 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.700127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.700141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.700212 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:13.700338 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.700823 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.701573 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.701763 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.701833 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.702018 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:13.702115 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:13.702192 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.702248 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.702270 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 15:15:13.702363 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.699895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44537 (local address 127.0.0.1:50470)
0925 15:15:13.699955 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.699968 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.700024 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.700219 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.700220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.700222 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:15:13.700270 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.700273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.700911 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.700912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.701489 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.701491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.701833 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.701834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.701894 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.701910 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:15:13.702180 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:15:13.702189 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.702243 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.702288 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 15:15:13.703665 9915 raft_consensus.cc:1261] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Refusing update from remote peer 1d323df359eb4dac9fa2454d33223def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:13.703701 9775 raft_consensus.cc:1261] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Refusing update from remote peer 1d323df359eb4dac9fa2454d33223def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:13.703953 10014 consensus_queue.cc:1035] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:13.704100 9991 consensus_queue.cc:1035] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def [LEADER]: Connected to new peer: Peer: permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:13.704665 9994 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: TxnStatusTablet state changed. Reason: New leader 1d323df359eb4dac9fa2454d33223def. Latest consensus state: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.704767 9994 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:13.705235 9991 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.705314 9991 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:15:13.705329 9992 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: TxnStatusTablet state changed. Reason: New leader 1d323df359eb4dac9fa2454d33223def. Latest consensus state: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.705415 9992 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:13.705478 9998 mvcc.cc:205] Tried to move back new op lower bound from 7333270788928036864 to 7333270788854665216. Current Snapshot: MvccSnapshot[applied={T|T < 7333270788928036864 or (T in {7333270788928036864})}]
I20260925 15:15:13.705804 9997 mvcc.cc:205] Tried to move back new op lower bound from 7333270788928036864 to 7333270788854665216. Current Snapshot: MvccSnapshot[applied={T|T < 7333270788928036864}]
I20260925 15:15:13.706110 9994 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.706187 9994 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P 032747bfd70a4c9ebb1d687200ba52af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:13.707206 10014 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.707317 10014 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P 1d323df359eb4dac9fa2454d33223def: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:15:13.707859 9992 tablet_replica.cc:439] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d323df359eb4dac9fa2454d33223def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d323df359eb4dac9fa2454d33223def" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44537 } } peers { permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 } } peers { permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 } } }
I20260925 15:15:13.707957 9992 tablet_replica.cc:442] T 2b210b1f697c4580bf9d62814b2baf65 P c32c1442908643e9986a970ab45d1b21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:13.710340 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.707075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.8.43.129:44537)
0925 15:15:13.707111 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.707112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.707258 (+ 146us) server_negotiation.cc:407] Connection header received
0925 15:15:13.707281 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.707282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.707293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.707340 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:13.708126 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.708845 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.709638 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.709945 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.710009 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.710127 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:13.710159 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:13.710210 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.710252 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.710270 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 15:15:13.715435 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.712844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:42675 (local address 127.8.43.129:44537)
0925 15:15:13.712892 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.712894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.713019 (+ 125us) server_negotiation.cc:407] Connection header received
0925 15:15:13.713063 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.713063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.713072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.713136 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:13.713226 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.713918 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.714657 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.714954 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.715020 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.715177 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:13.715205 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:13.715284 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.715334 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.715351 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:15:13.715509 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.712797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44537 (local address 127.8.43.129:42675)
0925 15:15:13.712948 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:13.712959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:13.713014 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:13.713138 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:13.713138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:13.713147 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:15:13.713191 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.713192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.713985 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.713986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.714581 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:13.714583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.715027 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.715028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:13.715090 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.715102 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:15:13.715285 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:15:13.715287 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:15:13.715331 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:15:13.715383 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260925 15:15:13.717906 9915 raft_consensus.cc:1261] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Refusing update from remote peer 1d323df359eb4dac9fa2454d33223def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:13.717949 9775 raft_consensus.cc:1261] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Refusing update from remote peer 1d323df359eb4dac9fa2454d33223def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:13.718137 10014 consensus_queue.cc:1035] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [LEADER]: Connected to new peer: Peer: permanent_uuid: "032747bfd70a4c9ebb1d687200ba52af" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 33599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:13.718264 10031 consensus_queue.cc:1035] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32c1442908643e9986a970ab45d1b21" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:13.719686 9999 mvcc.cc:205] Tried to move back new op lower bound from 7333270788987248640 to 7333270788516380672. Current Snapshot: MvccSnapshot[applied={T|T < 7333270788987248640}]
I20260925 15:15:13.732391 9478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:13.732493 9478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:13.733023 9478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deafd5bab0d24c99b5283b711fb2978c: Sending DeleteTablet for 3 replicas of tablet 27098dfe365f416782799fc02e328186
I20260925 15:15:13.733278 9755 tablet_service.cc:1514] Processing DeleteTablet for tablet 27098dfe365f416782799fc02e328186 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:13 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:43830
I20260925 15:15:13.733385 9895 tablet_service.cc:1514] Processing DeleteTablet for tablet 27098dfe365f416782799fc02e328186 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:13 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:55322
I20260925 15:15:13.733392 10006 tablet_replica.cc:330] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: stopping tablet replica
I20260925 15:15:13.733417 9615 tablet_service.cc:1514] Processing DeleteTablet for tablet 27098dfe365f416782799fc02e328186 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:13 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:50464
I20260925 15:15:13.733472 10006 raft_consensus.cc:2229] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:13.733493 10008 tablet_replica.cc:330] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: stopping tablet replica
I20260925 15:15:13.733489 10007 tablet_replica.cc:330] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: stopping tablet replica
I20260925 15:15:13.733515 10006 raft_consensus.cc:2258] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:13.733530 10008 raft_consensus.cc:2229] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:13.733538 10007 raft_consensus.cc:2229] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:13.733574 10007 raft_consensus.cc:2258] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:13.733603 10008 raft_consensus.cc:2258] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:13.733855 10007 ts_tablet_manager.cc:1905] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:13.734421 10008 ts_tablet_manager.cc:1905] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:13.735445 10007 ts_tablet_manager.cc:1918] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:13.735494 10007 log.cc:1196] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/27098dfe365f416782799fc02e328186
I20260925 15:15:13.735730 10007 ts_tablet_manager.cc:1939] T 27098dfe365f416782799fc02e328186 P c32c1442908643e9986a970ab45d1b21: Deleting consensus metadata
I20260925 15:15:13.736014 10008 ts_tablet_manager.cc:1918] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:13.736024 9464 catalog_manager.cc:4841] TS c32c1442908643e9986a970ab45d1b21 (127.8.43.131:41023): tablet 27098dfe365f416782799fc02e328186 (table test-table [id=fde508b93df046688adc832feb5b8582]) successfully deleted
I20260925 15:15:13.736059 10008 log.cc:1196] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/27098dfe365f416782799fc02e328186
I20260925 15:15:13.736256 10008 ts_tablet_manager.cc:1939] T 27098dfe365f416782799fc02e328186 P 1d323df359eb4dac9fa2454d33223def: Deleting consensus metadata
I20260925 15:15:13.736519 9462 catalog_manager.cc:4841] TS 1d323df359eb4dac9fa2454d33223def (127.8.43.129:44537): tablet 27098dfe365f416782799fc02e328186 (table test-table [id=fde508b93df046688adc832feb5b8582]) successfully deleted
I20260925 15:15:13.739449 10006 ts_tablet_manager.cc:1905] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:13.740932 10006 ts_tablet_manager.cc:1918] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:13.740999 10006 log.cc:1196] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/27098dfe365f416782799fc02e328186
I20260925 15:15:13.741202 10006 ts_tablet_manager.cc:1939] T 27098dfe365f416782799fc02e328186 P 032747bfd70a4c9ebb1d687200ba52af: Deleting consensus metadata
I20260925 15:15:13.741531 9465 catalog_manager.cc:4841] TS 032747bfd70a4c9ebb1d687200ba52af (127.8.43.130:33599): tablet 27098dfe365f416782799fc02e328186 (table test-table [id=fde508b93df046688adc832feb5b8582]) successfully deleted
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[9433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:13.745599 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.735563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.8.43.129:44537)
0925 15:15:13.736647 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.736649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.736658 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:13.736677 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.736678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.736688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.736755 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.737326 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.738060 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.738783 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.739088 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.739160 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.742555 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.742563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.742564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.742571 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.744432 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.744791 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.744794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.744795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.744861 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.745047 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.745048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.745048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.745208 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.745337 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.745375 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.745495 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065}
W20260925 15:15:13.745980 9675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50500
I20260925 15:15:13.753499 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:13.746907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.8.43.190:43797)
0925 15:15:13.746952 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:13.746954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:13.747054 (+ 100us) server_negotiation.cc:407] Connection header received
0925 15:15:13.747102 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:13.747103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:13.747113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:13.747171 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:13.747675 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.748450 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.749098 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:13.749400 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:13.749465 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:13.750680 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:13.750681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:13.750681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:13.750688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:13.752378 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.752750 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.752751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.752752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.752811 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:13.752993 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:13.752993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:13.752994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:13.753129 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:13.753230 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:13.753305 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:15:13.753379 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:15:13.753685 9478 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52306
I20260925 15:15:13.754387 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 9550
I20260925 15:15:13.760243 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 9690
I20260925 15:15:13.765923 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 9830
I20260925 15:15:13.770630 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 9446
2026-09-25T15:15:13Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2175 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[10044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[10044](info): set up 2 sockets
Sep 25 15:15:13 dist-test-slave-c5d6 krb5kdc[10044](info): commencing operation
krb5kdc: starting...
W20260925 15:15:14.802937 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:14 dist-test-slave-c5d6 krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:14.922737 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44965
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:15.012368 10057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:15.012624 10057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:15.012661 10057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:15.013816 10057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:15.013895 10057 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:15:15.013918 10057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:15.013942 10057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:15.013969 10057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:15.014001 10057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:15.015486 10057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44965
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:15.015764 10057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:15.016006 10057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:15.020911 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:15.021001 10057 server_base.cc:1034] running on GCE node
W20260925 15:15:15.020927 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:15.020970 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:15.021435 10057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:15.021745 10057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:15.022936 10057 hybrid_clock.cc:648] HybridClock initialized: now 1790349315022928 us; error 36 us; skew 500 ppm
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:15.025434 10057 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:15.025885 10057 webserver.cc:460] Webserver started at http://127.8.43.190:38063/ using document root <none> and password file <none>
I20260925 15:15:15.026108 10057 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:15.026183 10057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:15.026331 10057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:15.027483 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "c6cce67f67654782aa9b9816b8c0e3ad"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "4086b6ae11e13eac5901c86e0cb8a9a9"
server_key_iv: "0091ce5c83d4965434ec6a73264d1c2f"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.027850 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "c6cce67f67654782aa9b9816b8c0e3ad"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "4086b6ae11e13eac5901c86e0cb8a9a9"
server_key_iv: "0091ce5c83d4965434ec6a73264d1c2f"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.029300 10057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:15.030086 10072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.030308 10057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:15.030416 10057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "c6cce67f67654782aa9b9816b8c0e3ad"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "4086b6ae11e13eac5901c86e0cb8a9a9"
server_key_iv: "0091ce5c83d4965434ec6a73264d1c2f"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.030501 10057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:15.042234 10057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:15.044225 10057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:15.044399 10057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:15.048992 10057 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44965
I20260925 15:15:15.049032 10134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44965 every 8 connection(s)
I20260925 15:15:15.049440 10057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:15.050057 10135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:15.052320 10135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: Bootstrap starting.
I20260925 15:15:15.052881 10135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:15.053166 10135 log.cc:824] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:15.053825 10135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: No bootstrap required, opened a new log
I20260925 15:15:15.055963 10135 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cce67f67654782aa9b9816b8c0e3ad" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44965 } }
I20260925 15:15:15.056094 10135 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:15.056115 10135 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cce67f67654782aa9b9816b8c0e3ad, State: Initialized, Role: FOLLOWER
I20260925 15:15:15.056293 10135 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cce67f67654782aa9b9816b8c0e3ad" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44965 } }
I20260925 15:15:15.056372 10135 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:15.056416 10135 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:15.056463 10135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:15.056960 10135 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cce67f67654782aa9b9816b8c0e3ad" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44965 } }
I20260925 15:15:15.057044 10135 leader_election.cc:302] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6cce67f67654782aa9b9816b8c0e3ad; no voters:
I20260925 15:15:15.057221 10135 leader_election.cc:288] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:15.057291 10138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:15.057452 10135 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:15.057492 10138 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [term 1 LEADER]: Becoming Leader. State: Replica: c6cce67f67654782aa9b9816b8c0e3ad, State: Running, Role: LEADER
I20260925 15:15:15.057607 10138 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cce67f67654782aa9b9816b8c0e3ad" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44965 } }
I20260925 15:15:15.057982 10139 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6cce67f67654782aa9b9816b8c0e3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cce67f67654782aa9b9816b8c0e3ad" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44965 } } }
I20260925 15:15:15.057982 10140 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6cce67f67654782aa9b9816b8c0e3ad. Latest consensus state: current_term: 1 leader_uuid: "c6cce67f67654782aa9b9816b8c0e3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cce67f67654782aa9b9816b8c0e3ad" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44965 } } }
I20260925 15:15:15.058161 10140 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:15.058383 10139 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:15.058406 10147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:15.059000 10147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:15.059324 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 10057
I20260925 15:15:15.059409 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:15.059566 8366 external_mini_cluster.cc:1428] Setting key 6aac9c843bcb1486732be24426928383
I20260925 15:15:15.060583 10147 catalog_manager.cc:1347] Generated new cluster ID: d8f8e0f773aa4fba96f65c10aa49d1d4
I20260925 15:15:15.060650 10147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:15.073060 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.059991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.8.43.190:44965)
0925 15:15:15.060154 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.060165 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.060234 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:15:15.060377 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.060382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.060431 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.063381 (+ 2950us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.063490 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.064992 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.066046 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.066682 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.066793 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.069567 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.069571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.069571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.069583 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.071479 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.071872 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.071875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.071875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.071978 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.072179 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.072180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.072180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.072454 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.072574 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.072699 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.072821 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260925 15:15:15.074931 10147 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:15.075654 10147 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:15.081096 10147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: Generated new TSK 0
I20260925 15:15:15.081256 10147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:15.104130 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44965
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:15.184707 10161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:15.184927 10161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:15.184988 10161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:15.186098 10161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:15.186174 10161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:15.186209 10161 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:15:15.186235 10161 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:15:15.186264 10161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:15.187731 10161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:44965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:15.187983 10161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:15.188201 10161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:15.193176 10167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:15.193168 10169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:15.193430 10161 server_base.cc:1034] running on GCE node
W20260925 15:15:15.193167 10166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:15.193651 10161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:15.193847 10161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:15.195019 10161 hybrid_clock.cc:648] HybridClock initialized: now 1790349315194991 us; error 41 us; skew 500 ppm
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:15.197549 10161 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:15.197960 10161 webserver.cc:460] Webserver started at http://127.8.43.129:46065/ using document root <none> and password file <none>
I20260925 15:15:15.198177 10161 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:15.198244 10161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:15.198417 10161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:15.199517 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "98df7810e2ee46468799c563212402ab"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "1723a6994502f809dd16396651bb5711"
server_key_iv: "191e1e9905602b47157461682fd7cf7f"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.199867 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "98df7810e2ee46468799c563212402ab"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "1723a6994502f809dd16396651bb5711"
server_key_iv: "191e1e9905602b47157461682fd7cf7f"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.201205 10161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:15.201951 10176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.202126 10161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:15.202227 10161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "98df7810e2ee46468799c563212402ab"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "1723a6994502f809dd16396651bb5711"
server_key_iv: "191e1e9905602b47157461682fd7cf7f"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.202299 10161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:15.212739 10161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:15.214493 10161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:15.214646 10161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:15.215119 10161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:15.215180 10161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.215248 10161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:15.215287 10161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:15.222222 10161 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:33203
I20260925 15:15:15.222294 10291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:33203 every 8 connection(s)
I20260925 15:15:15.222584 10161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:15.225601 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.215703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:37083 (local address 127.8.43.190:44965)
0925 15:15:15.215744 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.215746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.215965 (+ 219us) server_negotiation.cc:407] Connection header received
0925 15:15:15.216489 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.216490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.216509 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.216560 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.217400 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.218215 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.219884 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.220225 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.220294 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.222712 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.222713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.222713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.222721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.224197 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.224641 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.224642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.224643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.224700 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.224959 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.224960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.224961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.225078 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.225152 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.225423 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.225509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:15.225782 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.215687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44965 (local address 127.8.43.129:37083)
0925 15:15:15.215837 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.215881 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.216368 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.216629 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.216632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.217013 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:15.217314 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.217320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.218317 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.218318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.219791 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.219794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.220318 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.220319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.220485 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.221105 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:15.221113 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:15.222606 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:15.224320 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.224327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.224329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.224551 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.224766 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.224767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.224767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.224885 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.225144 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:15.225150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:15.225289 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.225401 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.225559 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20260925 15:15:15.226294 10292 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44965
I20260925 15:15:15.226377 10292 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:15.226557 10292 heartbeater.cc:511] Master 127.8.43.190:44965 requested a full tablet report, sending...
I20260925 15:15:15.227032 10088 ts_manager.cc:194] Registered new tserver with Master: 98df7810e2ee46468799c563212402ab (127.8.43.129:33203)
I20260925 15:15:15.227814 10088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:37083
I20260925 15:15:15.229045 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 10161
I20260925 15:15:15.229159 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:15.229324 8366 external_mini_cluster.cc:1428] Setting key 3d098cb36f28d223f73c134c7b917d3b
I20260925 15:15:15.233397 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.227417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:48861 (local address 127.8.43.190:44965)
0925 15:15:15.227453 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.227455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.227579 (+ 124us) server_negotiation.cc:407] Connection header received
0925 15:15:15.227648 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.227649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.227659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.227712 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.228025 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.228706 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.229541 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.229745 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.229828 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.230696 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.230697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.230698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.230705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.232288 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.232697 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.232697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.232698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.232744 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.232944 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.232944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.232944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.233033 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.233135 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.233244 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.233317 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:15:15.233458 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.227351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44965 (local address 127.8.43.129:48861)
0925 15:15:15.227515 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.227524 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.227592 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.227718 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.227719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.227891 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:15.227964 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.227965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.228805 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.228806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.229444 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.229446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.229828 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.229829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.229908 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.230321 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:15.230326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:15.230592 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:15.232372 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.232373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.232373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.232624 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.232808 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.232808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.232809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.232879 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.233102 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:15.233103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:15.233147 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.233220 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.233318 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:15.254105 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44965
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:15.334618 10301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:15.334808 10301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:15.334832 10301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:15.335976 10301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:15.336014 10301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:15.336035 10301 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:15:15.336046 10301 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:15:15.336063 10301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:15.337435 10301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:44965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:15.337612 10301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:15.337783 10301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:15.342708 10307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:15.342715 10309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:15.342728 10306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:15.343030 10301 server_base.cc:1034] running on GCE node
I20260925 15:15:15.343303 10301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:15.343626 10301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:15.344766 10301 hybrid_clock.cc:648] HybridClock initialized: now 1790349315344753 us; error 27 us; skew 500 ppm
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:15.347263 10301 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:15.347754 10301 webserver.cc:460] Webserver started at http://127.8.43.130:45785/ using document root <none> and password file <none>
I20260925 15:15:15.347972 10301 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:15.348016 10301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:15.348153 10301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:15.349247 10301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "3d337488cfdb4b50ac7c89c7ad94ac22"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "94e30683d6e7ad3a5702d4a8ef8125b4"
server_key_iv: "d870b7bbff30bd63acd6a245f1b019b9"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.349609 10301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "3d337488cfdb4b50ac7c89c7ad94ac22"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "94e30683d6e7ad3a5702d4a8ef8125b4"
server_key_iv: "d870b7bbff30bd63acd6a245f1b019b9"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.350838 10301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:15.351575 10316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.351738 10301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.351835 10301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "3d337488cfdb4b50ac7c89c7ad94ac22"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "94e30683d6e7ad3a5702d4a8ef8125b4"
server_key_iv: "d870b7bbff30bd63acd6a245f1b019b9"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.351912 10301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:15.362732 10301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:15.364912 10301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:15.365064 10301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:15.365622 10301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:15.365669 10301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.365700 10301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:15.365718 10301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:15.372872 10301 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:38059
I20260925 15:15:15.372973 10431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:38059 every 8 connection(s)
I20260925 15:15:15.373261 10301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:15.375255 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.366010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:45655 (local address 127.8.43.190:44965)
0925 15:15:15.366083 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.366086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.366337 (+ 251us) server_negotiation.cc:407] Connection header received
0925 15:15:15.366806 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.366807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.366822 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.366907 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.367583 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.368087 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.369253 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.369517 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.369600 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.372190 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.372192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.372193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.372200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.373736 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.374244 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.374245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.374245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.374308 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.374599 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.374600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.374601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.374720 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.374795 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.375067 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.375142 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:15:15.375439 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.366003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44965 (local address 127.8.43.130:45655)
0925 15:15:15.366205 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.366246 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.366713 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.366986 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.366991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.367293 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:15.367485 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.367489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.368194 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.368196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.369163 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.369166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.369610 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.369611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.369726 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.370219 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:15.370226 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:15.372078 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:15.373827 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.373832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.373834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.374129 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.374400 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.374402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.374402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.374524 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.374788 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:15.374793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:15.374928 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.375030 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.375184 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260925 15:15:15.375869 10432 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44965
I20260925 15:15:15.375949 10432 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:15.376119 10432 heartbeater.cc:511] Master 127.8.43.190:44965 requested a full tablet report, sending...
I20260925 15:15:15.376437 10089 ts_manager.cc:194] Registered new tserver with Master: 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059)
I20260925 15:15:15.376832 10089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:45655
I20260925 15:15:15.378816 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 10301
I20260925 15:15:15.378916 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:15.379096 8366 external_mini_cluster.cc:1428] Setting key bec92ca9fccd87107d28fe82c5ab0f9e
I20260925 15:15:15.382390 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.376802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44965 (local address 127.8.43.130:40417)
0925 15:15:15.376949 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.376958 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.377017 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.377152 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.377153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.377319 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:15.377371 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.377372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.377993 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.377993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.378528 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.378529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.378916 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.378917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.379000 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.379445 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:15.379451 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:15.379828 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:15.381439 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.381439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.381440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.381596 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.381794 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.381795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.381795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.381842 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.382159 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:15.382160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:15.382196 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.382252 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.382322 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260925 15:15:15.382516 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.376852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:40417 (local address 127.8.43.190:44965)
0925 15:15:15.376896 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.376898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.377019 (+ 121us) server_negotiation.cc:407] Connection header received
0925 15:15:15.377086 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.377087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.377096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.377139 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.377443 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.377904 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.378622 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.378832 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.378913 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.379947 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.379949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.379949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.379955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.381341 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.381670 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.381671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.381671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.381732 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.381895 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.381896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.381896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.382089 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.382163 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.382296 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.382381 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:15.403788 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44965
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:15.488255 10441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:15.488580 10441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:15.488610 10441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:15.489759 10441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:15.489804 10441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:15.489835 10441 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:15:15.489849 10441 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:15:15.489868 10441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:15.491345 10441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:44965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:15.491529 10441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:15.491714 10441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:15.496843 10446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:15.497056 10449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:15.497144 10447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:15.497062 10441 server_base.cc:1034] running on GCE node
I20260925 15:15:15.497396 10441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:15.497665 10441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:15.498838 10441 hybrid_clock.cc:648] HybridClock initialized: now 1790349315498806 us; error 51 us; skew 500 ppm
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:15.501325 10441 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:15.501818 10441 webserver.cc:460] Webserver started at http://127.8.43.131:38297/ using document root <none> and password file <none>
I20260925 15:15:15.502060 10441 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:15.502127 10441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:15.502281 10441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:15.503414 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "01a4638187c148dcbcc0f0979135ce3c"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "891d77f74a1f1b87f1af8a8daa28425f"
server_key_iv: "82050504e00e3b4d295198e8bb52a03b"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.503768 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "01a4638187c148dcbcc0f0979135ce3c"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "891d77f74a1f1b87f1af8a8daa28425f"
server_key_iv: "82050504e00e3b4d295198e8bb52a03b"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.505108 10441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:15:15.505960 10456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.506140 10441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:15.506242 10441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "01a4638187c148dcbcc0f0979135ce3c"
format_stamp: "Formatted at 2026-09-25 15:15:15 on dist-test-slave-c5d6"
server_key: "891d77f74a1f1b87f1af8a8daa28425f"
server_key_iv: "82050504e00e3b4d295198e8bb52a03b"
server_key_version: "encryptionkey@0"
I20260925 15:15:15.506294 10441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:15.521411 10441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:15.523656 10441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:15.523777 10441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:15.524255 10441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:15.524324 10441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.524400 10441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:15.524431 10441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:15.531414 10441 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:35093
I20260925 15:15:15.531466 10571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:35093 every 8 connection(s)
I20260925 15:15:15.531839 10441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:15.534056 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.524803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:55423 (local address 127.8.43.190:44965)
0925 15:15:15.524846 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.524848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.525063 (+ 215us) server_negotiation.cc:407] Connection header received
0925 15:15:15.525556 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.525557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.525570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.525670 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.526594 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.527462 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.528491 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.528695 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.528778 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.531462 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.531463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.531464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.531471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.532935 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.533263 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.533264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.533265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.533312 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.533471 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.533471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.533472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.533602 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.533663 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.533876 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.533944 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:15.534183 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.524787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44965 (local address 127.8.43.131:55423)
0925 15:15:15.524955 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.524989 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.525459 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.525787 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.525790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.526146 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:15.526505 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.526512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.527553 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.527554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.528380 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.528382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.528819 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.528820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.528990 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.529594 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:15.529603 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:15.531344 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:15.532987 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.532991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.532993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.533198 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.533354 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.533354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.533354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.533434 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.533665 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:15.533669 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:15.533775 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.533866 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.533999 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260925 15:15:15.534610 10572 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44965
I20260925 15:15:15.534693 10572 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:15.534874 10572 heartbeater.cc:511] Master 127.8.43.190:44965 requested a full tablet report, sending...
I20260925 15:15:15.535240 10088 ts_manager.cc:194] Registered new tserver with Master: 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093)
I20260925 15:15:15.535593 10088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:55423
I20260925 15:15:15.539184 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 10441
I20260925 15:15:15.539317 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:15.539537 8366 external_mini_cluster.cc:1428] Setting key a3375ddd603531addb85a0a780026875
I20260925 15:15:15.540283 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:15:15.540576 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.535732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:51025 (local address 127.8.43.190:44965)
0925 15:15:15.535905 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.535907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.535913 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:15:15.535929 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.535929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.535937 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.535980 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.536300 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.536729 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.537354 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.537521 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.537567 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.538275 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.538276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.538276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.538282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.539500 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.539831 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.539832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.539833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.539895 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.540086 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.540087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.540087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.540191 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.540261 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.540425 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.540492 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260925 15:15:15.540592 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.535613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44965 (local address 127.8.43.131:51025)
0925 15:15:15.535804 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.535813 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.535853 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.535986 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.535986 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.536168 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:15.536231 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.536232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.536806 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.536807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.537290 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.537291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.537580 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.537581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.537632 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.537963 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:15.537967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:15.538206 (+ 239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:15.539560 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.539561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.539562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.539746 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.539960 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.539961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.539961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.540016 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.540254 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:15.540255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:15.540297 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.540398 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.540494 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:15.562212 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.555066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.8.43.190:44965)
0925 15:15:15.555129 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.555132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.555144 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:15:15.555254 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.555255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.555266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.555337 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.555908 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.556369 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.557187 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.557382 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.557482 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.559754 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.559755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.559756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.559764 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.561116 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.561457 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.561458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.561459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.561522 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.561715 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.561716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.561716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.561828 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.561934 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.562037 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.562110 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:15:15.563918 10088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:15.565455 10088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:15.571923 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.567552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:33203 (local address 127.0.0.1:50538)
0925 15:15:15.567829 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.567868 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.567946 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.568341 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.568344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.568346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:15.568421 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.568423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.569721 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.569722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.570667 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.570669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.571601 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.571602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.571688 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.571701 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.571776 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.571825 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260925 15:15:15.571928 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.567655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.8.43.129:33203)
0925 15:15:15.568125 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.568129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.568139 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:15:15.568182 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.568184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.568201 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.568271 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:15.568502 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.569638 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.570760 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.571516 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.571592 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.571650 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.571798 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.571833 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":79,"threads_started":1}
I20260925 15:15:15.572182 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.567747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35093 (local address 127.0.0.1:42008)
0925 15:15:15.568167 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.568177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.568227 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.568456 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.568457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.568459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:15.568499 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.568500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.569707 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.569707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.570995 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.570998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.571972 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.571973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.572035 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.572040 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.572087 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.572120 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":33,"thread_start_us":34,"threads_started":1}
I20260925 15:15:15.572479 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.567905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.8.43.131:35093)
0925 15:15:15.568125 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.568129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.568281 (+ 152us) server_negotiation.cc:407] Connection header received
0925 15:15:15.568318 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.568321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.568335 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.568401 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:15.568575 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.569621 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.571101 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.571828 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.571887 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.571938 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.572321 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.572363 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
I20260925 15:15:15.572721 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.567537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:38059 (local address 127.0.0.1:57008)
0925 15:15:15.567829 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.567874 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.567984 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.569254 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.569255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.569257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:15.569316 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.569318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.570381 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.570382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.571291 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.571294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.572390 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.572391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.572467 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.572481 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.572539 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.572658 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":63,"threads_started":1}
I20260925 15:15:15.572669 10226 tablet_service.cc:1467] Processing CreateTablet for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 (DEFAULT_TABLE table=test-table [id=39dd4389355e4832a03b69f30ca47e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:15.572836 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.567625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.8.43.130:38059)
0925 15:15:15.569093 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.569097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.569106 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:15.569135 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.569137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.569148 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.569255 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:15.569379 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.570293 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.571375 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.572311 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.572389 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.572438 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.572627 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.572669 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1438,"thread_start_us":90,"threads_started":1}
I20260925 15:15:15.573099 10226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:15.573015 10506 tablet_service.cc:1467] Processing CreateTablet for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 (DEFAULT_TABLE table=test-table [id=39dd4389355e4832a03b69f30ca47e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:15.573341 10506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:15.573518 10366 tablet_service.cc:1467] Processing CreateTablet for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 (DEFAULT_TABLE table=test-table [id=39dd4389355e4832a03b69f30ca47e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:15.573805 10366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:15.576143 10592 tablet_bootstrap.cc:492] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Bootstrap starting.
I20260925 15:15:15.576902 10592 tablet_bootstrap.cc:654] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:15.576954 10593 tablet_bootstrap.cc:492] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Bootstrap starting.
I20260925 15:15:15.577301 10594 tablet_bootstrap.cc:492] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Bootstrap starting.
I20260925 15:15:15.577270 10592 log.cc:824] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:15.577894 10593 tablet_bootstrap.cc:654] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:15.577945 10594 tablet_bootstrap.cc:654] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:15.578217 10594 log.cc:824] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:15.578238 10593 log.cc:824] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:15.578379 10592 tablet_bootstrap.cc:492] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: No bootstrap required, opened a new log
I20260925 15:15:15.578495 10592 ts_tablet_manager.cc:1397] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:15.578866 10594 tablet_bootstrap.cc:492] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: No bootstrap required, opened a new log
I20260925 15:15:15.578917 10594 ts_tablet_manager.cc:1397] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:15.580149 10592 raft_consensus.cc:348] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.580296 10592 raft_consensus.cc:374] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:15.580340 10592 raft_consensus.cc:729] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98df7810e2ee46468799c563212402ab, State: Initialized, Role: FOLLOWER
I20260925 15:15:15.580452 10592 consensus_queue.cc:260] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.580483 10594 raft_consensus.cc:348] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.580649 10594 raft_consensus.cc:374] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:15.580690 10594 raft_consensus.cc:729] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d337488cfdb4b50ac7c89c7ad94ac22, State: Initialized, Role: FOLLOWER
I20260925 15:15:15.580745 10592 ts_tablet_manager.cc:1428] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:15.580797 10594 consensus_queue.cc:260] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.580865 10292 heartbeater.cc:503] Master 127.8.43.190:44965 was elected leader, sending a full tablet report...
I20260925 15:15:15.581048 10594 ts_tablet_manager.cc:1428] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:15.581068 10432 heartbeater.cc:503] Master 127.8.43.190:44965 was elected leader, sending a full tablet report...
I20260925 15:15:15.581629 10593 tablet_bootstrap.cc:492] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: No bootstrap required, opened a new log
I20260925 15:15:15.581694 10593 ts_tablet_manager.cc:1397] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 15:15:15.583407 10593 raft_consensus.cc:348] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.583521 10593 raft_consensus.cc:374] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:15.583568 10593 raft_consensus.cc:729] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a4638187c148dcbcc0f0979135ce3c, State: Initialized, Role: FOLLOWER
I20260925 15:15:15.583671 10593 consensus_queue.cc:260] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.583912 10593 ts_tablet_manager.cc:1428] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:15.583982 10572 heartbeater.cc:503] Master 127.8.43.190:44965 was elected leader, sending a full tablet report...
W20260925 15:15:15.623844 10433 tablet.cc:2369] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:15:15.723326 10293 tablet.cc:2369] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:15.768286 10598 raft_consensus.cc:482] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:15.768548 10598 raft_consensus.cc:504] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.768997 10598 leader_election.cc:288] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059), 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093)
I20260925 15:15:15.772065 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.769340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:41535 (local address 127.8.43.131:35093)
0925 15:15:15.769391 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.769393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.769493 (+ 100us) server_negotiation.cc:407] Connection header received
0925 15:15:15.769557 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.769558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.769577 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.769656 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:15.769813 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.770308 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.771342 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.771712 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.771775 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.771783 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.771947 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.771992 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:15:15.772078 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.769227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35093 (local address 127.8.43.129:41535)
0925 15:15:15.769370 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.769381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.769493 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.769672 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.769672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.769675 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:15.769741 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.769743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.770407 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.770408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.771258 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.771261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.771796 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.771797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.771865 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.771869 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.771922 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.771956 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":56,"thread_start_us":78,"threads_started":1}
I20260925 15:15:15.772145 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.769304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:41493 (local address 127.8.43.130:38059)
0925 15:15:15.769353 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.769356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.769466 (+ 110us) server_negotiation.cc:407] Connection header received
0925 15:15:15.769556 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.769557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.769569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.769618 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:15.769886 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.770328 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.771396 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.771738 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.771831 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.771839 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.772032 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.772070 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:15:15.772403 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.769230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:38059 (local address 127.8.43.129:41493)
0925 15:15:15.769371 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.769384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.769459 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.769649 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.769650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.769653 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:15.769795 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.769798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.770439 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.770440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.771294 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.771298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.771864 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.771865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.771943 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.771947 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.772009 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.772049 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":32,"thread_start_us":67,"threads_started":1}
I20260925 15:15:15.772814 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00cb3a7d70e42c8aa98ab00b4af6cd1" candidate_uuid: "98df7810e2ee46468799c563212402ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a4638187c148dcbcc0f0979135ce3c" is_pre_election: true
I20260925 15:15:15.772820 10384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00cb3a7d70e42c8aa98ab00b4af6cd1" candidate_uuid: "98df7810e2ee46468799c563212402ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" is_pre_election: true
I20260925 15:15:15.772980 10526 raft_consensus.cc:2454] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98df7810e2ee46468799c563212402ab in term 0.
I20260925 15:15:15.772980 10384 raft_consensus.cc:2454] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98df7810e2ee46468799c563212402ab in term 0.
I20260925 15:15:15.773231 10177 leader_election.cc:302] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 98df7810e2ee46468799c563212402ab; no voters:
I20260925 15:15:15.773355 10598 raft_consensus.cc:2781] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:15.773411 10598 raft_consensus.cc:482] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:15.773442 10598 raft_consensus.cc:3037] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:15.774160 10598 raft_consensus.cc:504] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.774307 10598 leader_election.cc:288] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [CANDIDATE]: Term 1 election: Requested vote from peers 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059), 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093)
I20260925 15:15:15.774451 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00cb3a7d70e42c8aa98ab00b4af6cd1" candidate_uuid: "98df7810e2ee46468799c563212402ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a4638187c148dcbcc0f0979135ce3c"
I20260925 15:15:15.774466 10384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00cb3a7d70e42c8aa98ab00b4af6cd1" candidate_uuid: "98df7810e2ee46468799c563212402ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22"
I20260925 15:15:15.774533 10526 raft_consensus.cc:3037] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:15.774557 10384 raft_consensus.cc:3037] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:15.775377 10384 raft_consensus.cc:2454] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98df7810e2ee46468799c563212402ab in term 1.
I20260925 15:15:15.775349 10526 raft_consensus.cc:2454] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98df7810e2ee46468799c563212402ab in term 1.
I20260925 15:15:15.775529 10177 leader_election.cc:302] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 98df7810e2ee46468799c563212402ab; no voters:
I20260925 15:15:15.775619 10598 raft_consensus.cc:2781] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:15.775779 10598 raft_consensus.cc:686] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 1 LEADER]: Becoming Leader. State: Replica: 98df7810e2ee46468799c563212402ab, State: Running, Role: LEADER
I20260925 15:15:15.775875 10598 consensus_queue.cc:237] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.776711 10089 catalog_manager.cc:5495] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 98df7810e2ee46468799c563212402ab (127.8.43.129). New cstate: current_term: 1 leader_uuid: "98df7810e2ee46468799c563212402ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } health_report { overall_health: UNKNOWN } } }
W20260925 15:15:15.782490 10573 tablet.cc:2369] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:15:15 dist-test-slave-c5d6 krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:15.841216 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.828107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44965 (local address 127.0.0.1:42074)
0925 15:15:15.828302 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.828316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.828409 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.828555 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.828556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.828863 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:15.828910 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.828913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.829593 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.829594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.830202 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.830204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.830581 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.830581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.830629 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.837114 (+ 6485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:15.837120 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:15.838602 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:15.840200 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.840208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.840210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.840428 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.840607 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:15.840607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:15.840608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:15.840670 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:15.840896 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:15.840903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:15.840947 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.841004 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.841101 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":120,"threads_started":1}
I20260925 15:15:15.841226 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.828187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.8.43.190:44965)
0925 15:15:15.828262 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.828264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.828412 (+ 148us) server_negotiation.cc:407] Connection header received
0925 15:15:15.828478 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.828479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.828490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.828539 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:15.829005 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.829491 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.830299 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.830500 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.830556 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.838684 (+ 8128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:15.838685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:15.838685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:15.838692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:15.840124 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.840496 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.840497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.840497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.840545 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:15.840724 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:15.840724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:15.840725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:15.840824 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:15.840871 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.841031 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.841145 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":46}
I20260925 15:15:15.843374 10089 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:42074:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:15:15.843549 10089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:15.845265 10506 tablet_service.cc:1467] Processing CreateTablet for tablet 0f258baf286a4992a66534a7585ebaf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6633a82e27848ecbdfa9a9a9de034f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:15:15.845257 10226 tablet_service.cc:1467] Processing CreateTablet for tablet 0f258baf286a4992a66534a7585ebaf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6633a82e27848ecbdfa9a9a9de034f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:15:15.845422 10506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f258baf286a4992a66534a7585ebaf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:15.845468 10226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f258baf286a4992a66534a7585ebaf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:15.846688 10593 tablet_bootstrap.cc:492] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: Bootstrap starting.
I20260925 15:15:15.846819 10592 tablet_bootstrap.cc:492] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: Bootstrap starting.
I20260925 15:15:15.846947 10366 tablet_service.cc:1467] Processing CreateTablet for tablet 0f258baf286a4992a66534a7585ebaf3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6633a82e27848ecbdfa9a9a9de034f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:15:15.847103 10366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f258baf286a4992a66534a7585ebaf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:15.847271 10593 tablet_bootstrap.cc:654] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:15.847311 10592 tablet_bootstrap.cc:654] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:15.847882 10593 tablet_bootstrap.cc:492] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: No bootstrap required, opened a new log
I20260925 15:15:15.847920 10593 ts_tablet_manager.cc:1397] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:15.848014 10592 tablet_bootstrap.cc:492] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: No bootstrap required, opened a new log
I20260925 15:15:15.848031 10593 raft_consensus.cc:348] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.848059 10592 ts_tablet_manager.cc:1397] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:15.848078 10593 raft_consensus.cc:374] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:15.848093 10593 raft_consensus.cc:729] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a4638187c148dcbcc0f0979135ce3c, State: Initialized, Role: FOLLOWER
I20260925 15:15:15.848150 10593 consensus_queue.cc:260] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.848256 10593 ts_tablet_manager.cc:1428] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.848317 10600 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.848392 10600 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:15.848412 10592 raft_consensus.cc:348] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.848495 10592 raft_consensus.cc:374] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:15.848517 10592 raft_consensus.cc:729] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98df7810e2ee46468799c563212402ab, State: Initialized, Role: FOLLOWER
I20260925 15:15:15.848568 10592 consensus_queue.cc:260] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.848657 10592 ts_tablet_manager.cc:1428] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:15:15.848785 10598 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.848863 10594 tablet_bootstrap.cc:492] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: Bootstrap starting.
I20260925 15:15:15.848876 10598 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:15.849575 10594 tablet_bootstrap.cc:654] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:15.850278 10594 tablet_bootstrap.cc:492] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: No bootstrap required, opened a new log
I20260925 15:15:15.850347 10594 ts_tablet_manager.cc:1397] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:15.850549 10594 raft_consensus.cc:348] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.850636 10594 raft_consensus.cc:374] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:15.850672 10594 raft_consensus.cc:729] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d337488cfdb4b50ac7c89c7ad94ac22, State: Initialized, Role: FOLLOWER
I20260925 15:15:15.850726 10594 consensus_queue.cc:260] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.850889 10594 ts_tablet_manager.cc:1428] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:15.850960 10599 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.851053 10599 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:15.948583 10600 raft_consensus.cc:482] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:15.948700 10600 raft_consensus.cc:504] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.949132 10600 leader_election.cc:288] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98df7810e2ee46468799c563212402ab (127.8.43.129:33203), 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059)
I20260925 15:15:15.953285 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.949502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:56383 (local address 127.8.43.129:33203)
0925 15:15:15.949547 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.949549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.949627 (+ 78us) server_negotiation.cc:407] Connection header received
0925 15:15:15.949656 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.949657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.949671 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.949734 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:15.949878 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.950713 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.952281 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.952894 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.952972 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.952982 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.953171 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.953212 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:15:15.953285 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.949490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:45555 (local address 127.8.43.130:38059)
0925 15:15:15.949539 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.949542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.949553 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:15:15.949583 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.949584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.949594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.949685 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:15.949892 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.950698 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.952281 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.952886 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.952972 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.952982 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.953170 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.953213 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:15:15.953284 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.949326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:38059 (local address 127.8.43.131:45555)
0925 15:15:15.949380 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.949402 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.949465 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.949702 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.949703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.949720 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:15.949794 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.949796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.950791 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.950792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.952184 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.952188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.952977 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.952978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.953066 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.953076 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.953140 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.953188 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":110}
I20260925 15:15:15.953290 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.949337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:33203 (local address 127.8.43.131:56383)
0925 15:15:15.949568 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.949575 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.949613 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.949744 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.949745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.949746 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:15.949800 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.949801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.950791 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.950792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.952184 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.952188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.952992 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.952993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.953066 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.953076 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.953140 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.953187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":119,"thread_start_us":145,"threads_started":1}
I20260925 15:15:15.953532 10384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f258baf286a4992a66534a7585ebaf3" candidate_uuid: "01a4638187c148dcbcc0f0979135ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" is_pre_election: true
I20260925 15:15:15.953629 10384 raft_consensus.cc:2454] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a4638187c148dcbcc0f0979135ce3c in term 0.
I20260925 15:15:15.953840 10458 leader_election.cc:302] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 3d337488cfdb4b50ac7c89c7ad94ac22; no voters:
I20260925 15:15:15.953840 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f258baf286a4992a66534a7585ebaf3" candidate_uuid: "01a4638187c148dcbcc0f0979135ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98df7810e2ee46468799c563212402ab" is_pre_election: true
I20260925 15:15:15.953969 10246 raft_consensus.cc:2454] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a4638187c148dcbcc0f0979135ce3c in term 0.
I20260925 15:15:15.954058 10600 raft_consensus.cc:2781] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:15.954106 10600 raft_consensus.cc:482] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:15.954133 10600 raft_consensus.cc:3037] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:15.954936 10600 raft_consensus.cc:504] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.955118 10600 leader_election.cc:288] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [CANDIDATE]: Term 1 election: Requested vote from peers 98df7810e2ee46468799c563212402ab (127.8.43.129:33203), 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059)
I20260925 15:15:15.955338 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f258baf286a4992a66534a7585ebaf3" candidate_uuid: "01a4638187c148dcbcc0f0979135ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98df7810e2ee46468799c563212402ab"
I20260925 15:15:15.955348 10384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f258baf286a4992a66534a7585ebaf3" candidate_uuid: "01a4638187c148dcbcc0f0979135ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22"
I20260925 15:15:15.955420 10384 raft_consensus.cc:3037] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:15.955420 10246 raft_consensus.cc:3037] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:15.956183 10384 raft_consensus.cc:2454] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a4638187c148dcbcc0f0979135ce3c in term 1.
I20260925 15:15:15.956250 10246 raft_consensus.cc:2454] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a4638187c148dcbcc0f0979135ce3c in term 1.
I20260925 15:15:15.956369 10458 leader_election.cc:302] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 3d337488cfdb4b50ac7c89c7ad94ac22; no voters:
I20260925 15:15:15.956454 10600 raft_consensus.cc:2781] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:15.956610 10600 raft_consensus.cc:686] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [term 1 LEADER]: Becoming Leader. State: Replica: 01a4638187c148dcbcc0f0979135ce3c, State: Running, Role: LEADER
I20260925 15:15:15.956732 10600 consensus_queue.cc:237] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:15.957196 10615 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: TxnStatusTablet state changed. Reason: New leader 01a4638187c148dcbcc0f0979135ce3c. Latest consensus state: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.957293 10615 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:15:15.957511 10617 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:15:15.957621 10617 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:15:15.957631 10088 catalog_manager.cc:5495] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131). New cstate: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } health_report { overall_health: HEALTHY } } }
I20260925 15:15:15.966557 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.963397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35093 (local address 127.0.0.1:42020)
0925 15:15:15.963494 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.963507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.963553 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.963693 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.963694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.963695 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:15:15.963742 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.963744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.964335 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.964336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.965388 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.965391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.965980 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.965981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.966042 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.966057 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:15:15.966382 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:15:15.966412 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.966459 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.966497 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:15:15.966565 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.963515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.8.43.131:35093)
0925 15:15:15.963567 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.963569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.963578 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:15.963623 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.963624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.963637 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.963685 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:15.963807 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.964253 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.965499 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.965826 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.965914 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.966161 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:15.966295 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:15.966391 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.966464 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.966488 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:15:15.968211 10246 raft_consensus.cc:1261] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Refusing update from remote peer 01a4638187c148dcbcc0f0979135ce3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:15.968252 10384 raft_consensus.cc:1261] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Refusing update from remote peer 01a4638187c148dcbcc0f0979135ce3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:15.968506 10600 consensus_queue.cc:1035] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:15.968652 10615 consensus_queue.cc:1035] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:15.969662 10598 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: TxnStatusTablet state changed. Reason: New leader 01a4638187c148dcbcc0f0979135ce3c. Latest consensus state: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.969744 10598 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:15.969794 10599 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: TxnStatusTablet state changed. Reason: New leader 01a4638187c148dcbcc0f0979135ce3c. Latest consensus state: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.969903 10599 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:15.970911 10600 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.970997 10600 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:15:15.971603 10599 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.971691 10599 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 3d337488cfdb4b50ac7c89c7ad94ac22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:15.972205 10615 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.972297 10615 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 01a4638187c148dcbcc0f0979135ce3c: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:15:15.972549 10598 tablet_replica.cc:439] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } } }
I20260925 15:15:15.972635 10598 tablet_replica.cc:442] T 0f258baf286a4992a66534a7585ebaf3 P 98df7810e2ee46468799c563212402ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:15:15.973024 10620 mvcc.cc:205] Tried to move back new op lower bound from 7333270798203408384 to 7333270798159273984. Current Snapshot: MvccSnapshot[applied={T|T < 7333270798203408384 or (T in {7333270798203408384})}]
I20260925 15:15:15.973269 10622 mvcc.cc:205] Tried to move back new op lower bound from 7333270798203408384 to 7333270798159273984. Current Snapshot: MvccSnapshot[applied={T|T < 7333270798203408384 or (T in {7333270798203408384})}]
I20260925 15:15:15.976022 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.973428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.8.43.129:33203)
0925 15:15:15.973471 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.973473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.973481 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:15.973574 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.973575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.973592 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.973673 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:15.973912 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.974366 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.975128 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.975411 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.975489 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.975689 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:15.975795 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:15.975879 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.975927 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.975947 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:15:15.981699 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.978763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:34659 (local address 127.8.43.131:35093)
0925 15:15:15.978839 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.978842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.978995 (+ 153us) server_negotiation.cc:407] Connection header received
0925 15:15:15.979019 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.979020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.979032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.979093 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:15.979191 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.980050 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.980875 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.981206 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.981282 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.981459 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:15.981492 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:15.981551 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.981606 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.981625 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:15:15.981709 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.978719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35093 (local address 127.8.43.129:34659)
0925 15:15:15.978882 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.978893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.978956 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.979094 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.979095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.979096 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:15:15.979144 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.979145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.980126 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.980127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.980785 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.980787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.981286 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.981287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.981335 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.981367 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:15:15.981557 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:15:15.981558 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.981597 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.981632 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260925 15:15:15.984261 10384 raft_consensus.cc:1261] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Refusing update from remote peer 98df7810e2ee46468799c563212402ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:15.984402 10526 raft_consensus.cc:1261] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Refusing update from remote peer 98df7810e2ee46468799c563212402ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:15.984556 10598 consensus_queue.cc:1035] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:15.984684 10603 consensus_queue.cc:1035] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:15.986253 10621 mvcc.cc:205] Tried to move back new op lower bound from 7333270798269800448 to 7333270797418749952. Current Snapshot: MvccSnapshot[applied={T|T < 7333270798269800448}]
I20260925 15:15:15.994405 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.991602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:49341 (local address 127.8.43.129:33203)
0925 15:15:15.991641 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:15.991643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:15.991652 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:15.991713 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:15.991713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:15.991723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:15.991764 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:15.991906 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.992493 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.993716 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.993940 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.993992 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.994196 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:15.994223 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:15.994270 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:15.994321 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 15:15:15.994337 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:15.994446 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:15.991539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:33203 (local address 127.8.43.131:49341)
0925 15:15:15.991599 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:15.991611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:15.991657 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:15.991776 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:15.991777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:15.991778 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:15:15.991839 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.991841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.992572 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.992574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.993623 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:15.993625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:15.994022 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:15.994023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:15.994097 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:15.994113 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:15:15.994264 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:15:15.994265 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:15:15.994321 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:15:15.994365 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 15:15:16.003952 10089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:16.004074 10089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:16.004944 10089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: Sending DeleteTablet for 3 replicas of tablet c00cb3a7d70e42c8aa98ab00b4af6cd1
I20260925 15:15:16.005364 10366 tablet_service.cc:1514] Processing DeleteTablet for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:16 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:57008
I20260925 15:15:16.005635 10639 tablet_replica.cc:330] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: stopping tablet replica
I20260925 15:15:16.005661 10226 tablet_service.cc:1514] Processing DeleteTablet for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:16 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:50538
I20260925 15:15:16.005888 10639 raft_consensus.cc:2229] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:16.005919 10506 tablet_service.cc:1514] Processing DeleteTablet for tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:16 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:42008
I20260925 15:15:16.006052 10639 raft_consensus.cc:2258] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:16.006100 10089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:16.006244 10089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:16.006713 10639 ts_tablet_manager.cc:1905] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:16.007759 10226 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc179cc4c9e447ebefbc4e0d8895f59 (DEFAULT_TABLE table=test-table [id=ab9d4a41be914cfebec3e3434a2d2192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:16.007913 10226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc179cc4c9e447ebefbc4e0d8895f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:16.008704 10639 ts_tablet_manager.cc:1918] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:16.008845 10639 log.cc:1196] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/c00cb3a7d70e42c8aa98ab00b4af6cd1
I20260925 15:15:16.009172 10639 ts_tablet_manager.cc:1939] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting consensus metadata
I20260925 15:15:16.009382 10592 tablet_bootstrap.cc:492] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: Bootstrap starting.
I20260925 15:15:16.009755 10074 catalog_manager.cc:4841] TS 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059): tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 (table test-table [id=39dd4389355e4832a03b69f30ca47e15]) successfully deleted
I20260925 15:15:16.009881 10592 tablet_bootstrap.cc:654] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:16.010459 10366 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc179cc4c9e447ebefbc4e0d8895f59 (DEFAULT_TABLE table=test-table [id=ab9d4a41be914cfebec3e3434a2d2192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:16.010604 10366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc179cc4c9e447ebefbc4e0d8895f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:16.010629 10592 tablet_bootstrap.cc:492] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: No bootstrap required, opened a new log
I20260925 15:15:16.010673 10592 ts_tablet_manager.cc:1397] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:16.010814 10592 raft_consensus.cc:348] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.010882 10592 raft_consensus.cc:374] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:16.010903 10592 raft_consensus.cc:729] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98df7810e2ee46468799c563212402ab, State: Initialized, Role: FOLLOWER
I20260925 15:15:16.010946 10592 consensus_queue.cc:260] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.011032 10592 ts_tablet_manager.cc:1428] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:16.011909 10594 tablet_bootstrap.cc:492] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: Bootstrap starting.
I20260925 15:15:16.012303 10594 tablet_bootstrap.cc:654] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:16.012548 10506 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc179cc4c9e447ebefbc4e0d8895f59 (DEFAULT_TABLE table=test-table [id=ab9d4a41be914cfebec3e3434a2d2192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:16.012677 10506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc179cc4c9e447ebefbc4e0d8895f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:16.012995 10594 tablet_bootstrap.cc:492] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: No bootstrap required, opened a new log
I20260925 15:15:16.013042 10594 ts_tablet_manager.cc:1397] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:16.013175 10594 raft_consensus.cc:348] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.013235 10594 raft_consensus.cc:374] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:16.013255 10594 raft_consensus.cc:729] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d337488cfdb4b50ac7c89c7ad94ac22, State: Initialized, Role: FOLLOWER
I20260925 15:15:16.013307 10594 consensus_queue.cc:260] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.013386 10594 ts_tablet_manager.cc:1428] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:16.014075 10593 tablet_bootstrap.cc:492] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: Bootstrap starting.
I20260925 15:15:16.014497 10593 tablet_bootstrap.cc:654] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:16.014858 10640 tablet_replica.cc:330] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: stopping tablet replica
I20260925 15:15:16.015012 10640 raft_consensus.cc:2229] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:16.015127 10640 raft_consensus.cc:2258] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:16.015298 10641 tablet_replica.cc:330] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: stopping tablet replica
I20260925 15:15:16.015408 10641 raft_consensus.cc:2229] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:16.015457 10641 raft_consensus.cc:2258] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:16.016191 10641 ts_tablet_manager.cc:1905] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:16.018119 10640 ts_tablet_manager.cc:1905] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:16.018258 10641 ts_tablet_manager.cc:1918] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:16.018365 10641 log.cc:1196] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/c00cb3a7d70e42c8aa98ab00b4af6cd1
I20260925 15:15:16.018684 10641 ts_tablet_manager.cc:1939] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 01a4638187c148dcbcc0f0979135ce3c: Deleting consensus metadata
I20260925 15:15:16.019147 10073 catalog_manager.cc:4841] TS 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093): tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 (table test-table [id=39dd4389355e4832a03b69f30ca47e15]) successfully deleted
I20260925 15:15:16.020041 10640 ts_tablet_manager.cc:1918] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:16.020123 10640 log.cc:1196] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/c00cb3a7d70e42c8aa98ab00b4af6cd1
I20260925 15:15:16.020401 10640 ts_tablet_manager.cc:1939] T c00cb3a7d70e42c8aa98ab00b4af6cd1 P 98df7810e2ee46468799c563212402ab: Deleting consensus metadata
I20260925 15:15:16.020969 10593 tablet_bootstrap.cc:492] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: No bootstrap required, opened a new log
I20260925 15:15:16.021035 10593 ts_tablet_manager.cc:1397] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 15:15:16.021179 10593 raft_consensus.cc:348] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.021263 10593 raft_consensus.cc:374] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:16.021291 10593 raft_consensus.cc:729] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a4638187c148dcbcc0f0979135ce3c, State: Initialized, Role: FOLLOWER
I20260925 15:15:16.021333 10593 consensus_queue.cc:260] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.021435 10593 ts_tablet_manager.cc:1428] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:16.021816 10073 catalog_manager.cc:4841] TS 98df7810e2ee46468799c563212402ab (127.8.43.129:33203): tablet c00cb3a7d70e42c8aa98ab00b4af6cd1 (table test-table [id=39dd4389355e4832a03b69f30ca47e15]) successfully deleted
I20260925 15:15:16.105118 10635 raft_consensus.cc:482] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:16.105226 10635 raft_consensus.cc:504] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.105389 10635 leader_election.cc:288] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059), 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093)
I20260925 15:15:16.105674 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc179cc4c9e447ebefbc4e0d8895f59" candidate_uuid: "98df7810e2ee46468799c563212402ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a4638187c148dcbcc0f0979135ce3c" is_pre_election: true
I20260925 15:15:16.105760 10526 raft_consensus.cc:2454] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98df7810e2ee46468799c563212402ab in term 0.
I20260925 15:15:16.105706 10384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc179cc4c9e447ebefbc4e0d8895f59" candidate_uuid: "98df7810e2ee46468799c563212402ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" is_pre_election: true
I20260925 15:15:16.105818 10384 raft_consensus.cc:2454] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98df7810e2ee46468799c563212402ab in term 0.
I20260925 15:15:16.105957 10177 leader_election.cc:302] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 98df7810e2ee46468799c563212402ab; no voters:
I20260925 15:15:16.106060 10635 raft_consensus.cc:2781] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:16.106093 10635 raft_consensus.cc:482] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:16.106112 10635 raft_consensus.cc:3037] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:16.107052 10635 raft_consensus.cc:504] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.107195 10635 leader_election.cc:288] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [CANDIDATE]: Term 1 election: Requested vote from peers 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059), 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093)
I20260925 15:15:16.107381 10384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc179cc4c9e447ebefbc4e0d8895f59" candidate_uuid: "98df7810e2ee46468799c563212402ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22"
I20260925 15:15:16.107414 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc179cc4c9e447ebefbc4e0d8895f59" candidate_uuid: "98df7810e2ee46468799c563212402ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a4638187c148dcbcc0f0979135ce3c"
I20260925 15:15:16.107453 10384 raft_consensus.cc:3037] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:16.107484 10526 raft_consensus.cc:3037] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:16.108239 10526 raft_consensus.cc:2454] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98df7810e2ee46468799c563212402ab in term 1.
I20260925 15:15:16.108253 10384 raft_consensus.cc:2454] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98df7810e2ee46468799c563212402ab in term 1.
I20260925 15:15:16.108424 10177 leader_election.cc:302] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 98df7810e2ee46468799c563212402ab; no voters:
I20260925 15:15:16.108570 10635 raft_consensus.cc:2781] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:16.108616 10635 raft_consensus.cc:686] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 1 LEADER]: Becoming Leader. State: Replica: 98df7810e2ee46468799c563212402ab, State: Running, Role: LEADER
I20260925 15:15:16.108669 10635 consensus_queue.cc:237] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:16.109217 10088 catalog_manager.cc:5495] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 98df7810e2ee46468799c563212402ab (127.8.43.129). New cstate: current_term: 1 leader_uuid: "98df7810e2ee46468799c563212402ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } health_report { overall_health: UNKNOWN } } }
I20260925 15:15:16.138218 10526 raft_consensus.cc:1261] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Refusing update from remote peer 98df7810e2ee46468799c563212402ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:16.138248 10384 raft_consensus.cc:1261] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Refusing update from remote peer 98df7810e2ee46468799c563212402ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:16.138440 10635 consensus_queue.cc:1035] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:16.138541 10603 consensus_queue.cc:1035] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:15:16.142696 10621 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:15:17.659844 10089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:17.659982 10089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:17.660712 10089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: Sending DeleteTablet for 3 replicas of tablet 1dc179cc4c9e447ebefbc4e0d8895f59
I20260925 15:15:17.661002 10226 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc179cc4c9e447ebefbc4e0d8895f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:17 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:50538
I20260925 15:15:17.660987 10366 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc179cc4c9e447ebefbc4e0d8895f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:17 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:57008
I20260925 15:15:17.661060 10506 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc179cc4c9e447ebefbc4e0d8895f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:17 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:42008
I20260925 15:15:17.661217 10669 tablet_replica.cc:330] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: stopping tablet replica
I20260925 15:15:17.661237 10671 tablet_replica.cc:330] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: stopping tablet replica
I20260925 15:15:17.661279 10669 raft_consensus.cc:2229] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:17.661285 10671 raft_consensus.cc:2229] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:17.661319 10671 raft_consensus.cc:2258] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:17.661597 10669 raft_consensus.cc:2258] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:17.661666 10671 ts_tablet_manager.cc:1905] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:17.661844 10669 ts_tablet_manager.cc:1905] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:17.663399 10670 tablet_replica.cc:330] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: stopping tablet replica
I20260925 15:15:17.663436 10669 ts_tablet_manager.cc:1918] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:15:17.663483 10669 log.cc:1196] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/1dc179cc4c9e447ebefbc4e0d8895f59
I20260925 15:15:17.663580 10670 raft_consensus.cc:2229] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:17.663709 10671 ts_tablet_manager.cc:1918] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:15:17.663729 10669 ts_tablet_manager.cc:1939] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting consensus metadata
I20260925 15:15:17.663852 10670 raft_consensus.cc:2258] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:17.664168 10670 ts_tablet_manager.cc:1905] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:17.664345 10074 catalog_manager.cc:4841] TS 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059): tablet 1dc179cc4c9e447ebefbc4e0d8895f59 (table test-table [id=ab9d4a41be914cfebec3e3434a2d2192]) successfully deleted
I20260925 15:15:17.665462 10670 ts_tablet_manager.cc:1918] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:15:17.665522 10670 log.cc:1196] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/1dc179cc4c9e447ebefbc4e0d8895f59
I20260925 15:15:17.665742 10670 ts_tablet_manager.cc:1939] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 98df7810e2ee46468799c563212402ab: Deleting consensus metadata
I20260925 15:15:17.666064 10073 catalog_manager.cc:4841] TS 98df7810e2ee46468799c563212402ab (127.8.43.129:33203): tablet 1dc179cc4c9e447ebefbc4e0d8895f59 (table test-table [id=ab9d4a41be914cfebec3e3434a2d2192]) successfully deleted
I20260925 15:15:17.666345 10671 log.cc:1196] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/1dc179cc4c9e447ebefbc4e0d8895f59
I20260925 15:15:17.666580 10671 ts_tablet_manager.cc:1939] T 1dc179cc4c9e447ebefbc4e0d8895f59 P 01a4638187c148dcbcc0f0979135ce3c: Deleting consensus metadata
I20260925 15:15:17.666895 10073 catalog_manager.cc:4841] TS 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093): tablet 1dc179cc4c9e447ebefbc4e0d8895f59 (table test-table [id=ab9d4a41be914cfebec3e3434a2d2192]) successfully deleted
Sep 25 15:15:17 dist-test-slave-c5d6 krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:15:17 dist-test-slave-c5d6 krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:17.684204 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:17.677248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.8.43.190:44965)
0925 15:15:17.677398 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:17.677400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:17.677410 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:15:17.677482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:17.677482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:17.677492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:17.677553 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:17.678129 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:17.678636 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:17.679435 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:17.679624 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:17.679676 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:17.681841 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:17.681842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:17.681843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:17.681849 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:17.683269 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:17.683595 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:17.683596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:17.683596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:17.683645 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:17.683806 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:17.683806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:17.683806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:17.683914 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:17.683986 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:17.684083 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 15:15:17.684139 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":97,"threads_started":1}
I20260925 15:15:17.685649 10089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:17.685813 10089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:17.687042 10226 tablet_service.cc:1467] Processing CreateTablet for tablet b6f68ffc0b5f40e9a7252a20d2546780 (DEFAULT_TABLE table=test-table [id=55cbb450ed094147a2c18dbf24166ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:17.687071 10366 tablet_service.cc:1467] Processing CreateTablet for tablet b6f68ffc0b5f40e9a7252a20d2546780 (DEFAULT_TABLE table=test-table [id=55cbb450ed094147a2c18dbf24166ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:17.687080 10506 tablet_service.cc:1467] Processing CreateTablet for tablet b6f68ffc0b5f40e9a7252a20d2546780 (DEFAULT_TABLE table=test-table [id=55cbb450ed094147a2c18dbf24166ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:17.687182 10226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f68ffc0b5f40e9a7252a20d2546780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:17.687182 10366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f68ffc0b5f40e9a7252a20d2546780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:17.687217 10506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6f68ffc0b5f40e9a7252a20d2546780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:17.688449 10680 tablet_bootstrap.cc:492] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: Bootstrap starting.
I20260925 15:15:17.688683 10681 tablet_bootstrap.cc:492] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: Bootstrap starting.
I20260925 15:15:17.688819 10680 tablet_bootstrap.cc:654] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:17.689086 10681 tablet_bootstrap.cc:654] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:17.689121 10682 tablet_bootstrap.cc:492] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: Bootstrap starting.
I20260925 15:15:17.689435 10680 tablet_bootstrap.cc:492] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: No bootstrap required, opened a new log
I20260925 15:15:17.689502 10680 ts_tablet_manager.cc:1397] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:17.689512 10682 tablet_bootstrap.cc:654] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:17.689692 10681 tablet_bootstrap.cc:492] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: No bootstrap required, opened a new log
I20260925 15:15:17.689672 10680 raft_consensus.cc:348] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.689762 10680 raft_consensus.cc:374] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:17.689786 10680 raft_consensus.cc:729] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a4638187c148dcbcc0f0979135ce3c, State: Initialized, Role: FOLLOWER
I20260925 15:15:17.689776 10681 ts_tablet_manager.cc:1397] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:17.689860 10680 consensus_queue.cc:260] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.689904 10681 raft_consensus.cc:348] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.689977 10681 raft_consensus.cc:374] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:17.689996 10680 ts_tablet_manager.cc:1428] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:17.690039 10681 raft_consensus.cc:729] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98df7810e2ee46468799c563212402ab, State: Initialized, Role: FOLLOWER
I20260925 15:15:17.690111 10681 consensus_queue.cc:260] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.690276 10681 ts_tablet_manager.cc:1428] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:17.690327 10682 tablet_bootstrap.cc:492] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: No bootstrap required, opened a new log
I20260925 15:15:17.690387 10682 ts_tablet_manager.cc:1397] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:17.690680 10682 raft_consensus.cc:348] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.690795 10682 raft_consensus.cc:374] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:17.690827 10682 raft_consensus.cc:729] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d337488cfdb4b50ac7c89c7ad94ac22, State: Initialized, Role: FOLLOWER
I20260925 15:15:17.690878 10682 consensus_queue.cc:260] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.691047 10682 ts_tablet_manager.cc:1428] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:17.934809 10655 raft_consensus.cc:482] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:17.934939 10655 raft_consensus.cc:504] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.935163 10655 leader_election.cc:288] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98df7810e2ee46468799c563212402ab (127.8.43.129:33203), 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059)
I20260925 15:15:17.935416 10386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f68ffc0b5f40e9a7252a20d2546780" candidate_uuid: "01a4638187c148dcbcc0f0979135ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" is_pre_election: true
I20260925 15:15:17.935417 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f68ffc0b5f40e9a7252a20d2546780" candidate_uuid: "01a4638187c148dcbcc0f0979135ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98df7810e2ee46468799c563212402ab" is_pre_election: true
I20260925 15:15:17.935516 10246 raft_consensus.cc:2454] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a4638187c148dcbcc0f0979135ce3c in term 0.
I20260925 15:15:17.935516 10386 raft_consensus.cc:2454] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a4638187c148dcbcc0f0979135ce3c in term 0.
I20260925 15:15:17.935698 10458 leader_election.cc:302] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 3d337488cfdb4b50ac7c89c7ad94ac22; no voters:
I20260925 15:15:17.935822 10655 raft_consensus.cc:2781] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:17.935881 10655 raft_consensus.cc:482] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:17.935910 10655 raft_consensus.cc:3037] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:17.936645 10655 raft_consensus.cc:504] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.936815 10655 leader_election.cc:288] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [CANDIDATE]: Term 1 election: Requested vote from peers 98df7810e2ee46468799c563212402ab (127.8.43.129:33203), 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059)
I20260925 15:15:17.936985 10386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f68ffc0b5f40e9a7252a20d2546780" candidate_uuid: "01a4638187c148dcbcc0f0979135ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22"
I20260925 15:15:17.936996 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f68ffc0b5f40e9a7252a20d2546780" candidate_uuid: "01a4638187c148dcbcc0f0979135ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98df7810e2ee46468799c563212402ab"
I20260925 15:15:17.937065 10246 raft_consensus.cc:3037] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:17.937065 10386 raft_consensus.cc:3037] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:17.937819 10246 raft_consensus.cc:2454] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a4638187c148dcbcc0f0979135ce3c in term 1.
I20260925 15:15:17.937819 10386 raft_consensus.cc:2454] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a4638187c148dcbcc0f0979135ce3c in term 1.
I20260925 15:15:17.938006 10458 leader_election.cc:302] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 3d337488cfdb4b50ac7c89c7ad94ac22; no voters:
I20260925 15:15:17.938118 10655 raft_consensus.cc:2781] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:17.938174 10655 raft_consensus.cc:686] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 1 LEADER]: Becoming Leader. State: Replica: 01a4638187c148dcbcc0f0979135ce3c, State: Running, Role: LEADER
I20260925 15:15:17.938208 10655 consensus_queue.cc:237] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:17.938854 10088 catalog_manager.cc:5495] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131). New cstate: current_term: 1 leader_uuid: "01a4638187c148dcbcc0f0979135ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } health_report { overall_health: HEALTHY } } }
I20260925 15:15:17.975449 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:17.973159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.8.43.131:35093)
0925 15:15:17.973310 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:17.973313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:17.973320 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:15:17.973338 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:17.973339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:17.973348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:17.973410 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:17.973513 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:17.973991 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:17.974794 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:17.974996 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:17.975048 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:17.975175 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:17.975205 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:17.975268 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:17.975362 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 15:15:17.975382 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260925 15:15:17.980001 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:17.977772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:38337 (local address 127.8.43.131:35093)
0925 15:15:17.977838 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:17.977840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:17.977948 (+ 108us) server_negotiation.cc:407] Connection header received
0925 15:15:17.978001 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:17.978001 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:17.978010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:17.978058 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:17.978172 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:17.978627 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:17.979360 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:17.979551 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:17.979622 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:17.979753 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:17.979782 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:17.979850 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:17.979900 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 15:15:17.979924 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:15:17.980005 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:17.977701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35093 (local address 127.8.43.131:38337)
0925 15:15:17.977862 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:17.977872 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:17.977948 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:17.978059 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:17.978060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:17.978061 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:15:17.978100 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:17.978101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:17.978706 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:17.978707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:17.979272 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:17.979274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:17.979621 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:17.979622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:17.979670 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:17.979673 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:15:17.979849 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:15:17.979851 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:15:17.979898 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:15:17.979936 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260925 15:15:17.982074 10386 raft_consensus.cc:1261] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Refusing update from remote peer 01a4638187c148dcbcc0f0979135ce3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:17.982107 10246 raft_consensus.cc:1261] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Refusing update from remote peer 01a4638187c148dcbcc0f0979135ce3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:17.982342 10650 consensus_queue.cc:1035] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:17.982426 10600 consensus_queue.cc:1035] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:17.993239 10088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:17.993369 10088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:17.993990 10088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: Sending DeleteTablet for 3 replicas of tablet b6f68ffc0b5f40e9a7252a20d2546780
I20260925 15:15:17.994215 10226 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f68ffc0b5f40e9a7252a20d2546780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:17 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:50538
I20260925 15:15:17.994246 10506 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f68ffc0b5f40e9a7252a20d2546780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:17 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:42008
I20260925 15:15:17.994266 10366 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f68ffc0b5f40e9a7252a20d2546780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:17 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:57008
I20260925 15:15:17.994308 10670 tablet_replica.cc:330] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: stopping tablet replica
I20260925 15:15:17.994341 10671 tablet_replica.cc:330] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: stopping tablet replica
I20260925 15:15:17.994351 10670 raft_consensus.cc:2229] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:17.994395 10671 raft_consensus.cc:2229] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:17.994383 10670 raft_consensus.cc:2258] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:17.994488 10671 raft_consensus.cc:2258] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:17.994560 10669 tablet_replica.cc:330] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: stopping tablet replica
I20260925 15:15:17.994613 10669 raft_consensus.cc:2229] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:17.994652 10669 raft_consensus.cc:2258] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:17.994788 10670 ts_tablet_manager.cc:1905] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:17.995064 10669 ts_tablet_manager.cc:1905] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:17.995146 10671 ts_tablet_manager.cc:1905] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:17.996420 10670 ts_tablet_manager.cc:1918] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:17.996486 10670 log.cc:1196] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/b6f68ffc0b5f40e9a7252a20d2546780
I20260925 15:15:17.996706 10670 ts_tablet_manager.cc:1939] T b6f68ffc0b5f40e9a7252a20d2546780 P 98df7810e2ee46468799c563212402ab: Deleting consensus metadata
I20260925 15:15:17.996711 10671 ts_tablet_manager.cc:1918] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:17.996771 10671 log.cc:1196] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/b6f68ffc0b5f40e9a7252a20d2546780
I20260925 15:15:17.996999 10671 ts_tablet_manager.cc:1939] T b6f68ffc0b5f40e9a7252a20d2546780 P 01a4638187c148dcbcc0f0979135ce3c: Deleting consensus metadata
I20260925 15:15:17.997144 10073 catalog_manager.cc:4841] TS 98df7810e2ee46468799c563212402ab (127.8.43.129:33203): tablet b6f68ffc0b5f40e9a7252a20d2546780 (table test-table [id=55cbb450ed094147a2c18dbf24166ab8]) successfully deleted
I20260925 15:15:17.997200 10669 ts_tablet_manager.cc:1918] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:15:17.997246 10669 log.cc:1196] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/b6f68ffc0b5f40e9a7252a20d2546780
I20260925 15:15:17.997491 10669 ts_tablet_manager.cc:1939] T b6f68ffc0b5f40e9a7252a20d2546780 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting consensus metadata
I20260925 15:15:17.997812 10073 catalog_manager.cc:4841] TS 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093): tablet b6f68ffc0b5f40e9a7252a20d2546780 (table test-table [id=55cbb450ed094147a2c18dbf24166ab8]) successfully deleted
I20260925 15:15:17.997812 10074 catalog_manager.cc:4841] TS 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059): tablet b6f68ffc0b5f40e9a7252a20d2546780 (table test-table [id=55cbb450ed094147a2c18dbf24166ab8]) successfully deleted
I20260925 15:15:17.998281 10088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:17.998453 10088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:17.999902 10226 tablet_service.cc:1467] Processing CreateTablet for tablet f2b46358f2874b47a309c51c21d24be8 (DEFAULT_TABLE table=test-table [id=cb51dfcf72af40f49d8bfdd51f04c6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:17.999909 10366 tablet_service.cc:1467] Processing CreateTablet for tablet f2b46358f2874b47a309c51c21d24be8 (DEFAULT_TABLE table=test-table [id=cb51dfcf72af40f49d8bfdd51f04c6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:18.000020 10226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b46358f2874b47a309c51c21d24be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:18.000048 10366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b46358f2874b47a309c51c21d24be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:18.000973 10682 tablet_bootstrap.cc:492] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: Bootstrap starting.
I20260925 15:15:18.001179 10681 tablet_bootstrap.cc:492] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab: Bootstrap starting.
I20260925 15:15:18.001263 10506 tablet_service.cc:1467] Processing CreateTablet for tablet f2b46358f2874b47a309c51c21d24be8 (DEFAULT_TABLE table=test-table [id=cb51dfcf72af40f49d8bfdd51f04c6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:18.001360 10506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b46358f2874b47a309c51c21d24be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:18.001590 10682 tablet_bootstrap.cc:654] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:18.001590 10681 tablet_bootstrap.cc:654] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:18.002233 10681 tablet_bootstrap.cc:492] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab: No bootstrap required, opened a new log
I20260925 15:15:18.002233 10682 tablet_bootstrap.cc:492] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: No bootstrap required, opened a new log
I20260925 15:15:18.002281 10681 ts_tablet_manager.cc:1397] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:18.002281 10682 ts_tablet_manager.cc:1397] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:18.002401 10681 raft_consensus.cc:348] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.002478 10681 raft_consensus.cc:374] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:18.002450 10682 raft_consensus.cc:348] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.002501 10681 raft_consensus.cc:729] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98df7810e2ee46468799c563212402ab, State: Initialized, Role: FOLLOWER
I20260925 15:15:18.002521 10682 raft_consensus.cc:374] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:18.002549 10682 raft_consensus.cc:729] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d337488cfdb4b50ac7c89c7ad94ac22, State: Initialized, Role: FOLLOWER
I20260925 15:15:18.002542 10680 tablet_bootstrap.cc:492] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: Bootstrap starting.
I20260925 15:15:18.002553 10681 consensus_queue.cc:260] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.002595 10682 consensus_queue.cc:260] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.002657 10681 ts_tablet_manager.cc:1428] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:18.002684 10682 ts_tablet_manager.cc:1428] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:18.002863 10680 tablet_bootstrap.cc:654] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:18.003417 10680 tablet_bootstrap.cc:492] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: No bootstrap required, opened a new log
I20260925 15:15:18.003466 10680 ts_tablet_manager.cc:1397] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:18.003569 10680 raft_consensus.cc:348] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.003647 10680 raft_consensus.cc:374] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:18.003674 10680 raft_consensus.cc:729] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a4638187c148dcbcc0f0979135ce3c, State: Initialized, Role: FOLLOWER
I20260925 15:15:18.003727 10680 consensus_queue.cc:260] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.003811 10680 ts_tablet_manager.cc:1428] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:18.028797 10686 raft_consensus.cc:482] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:18.028951 10686 raft_consensus.cc:504] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.029330 10686 leader_election.cc:288] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98df7810e2ee46468799c563212402ab (127.8.43.129:33203), 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093)
I20260925 15:15:18.033517 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.029805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:38029 (local address 127.8.43.131:35093)
0925 15:15:18.029854 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:18.029856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:18.029975 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:15:18.030040 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:18.030041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:18.030061 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:18.030117 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:18.030318 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.030898 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.032548 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.033135 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.033214 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.033223 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:18.033406 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:15:18.033444 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:15:18.033558 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.029521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:33203 (local address 127.8.43.130:35669)
0925 15:15:18.029635 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:18.029650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:18.029730 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:18.030068 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:18.030069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:18.030071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:18.030157 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:18.030159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.031177 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.031178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:18.032440 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:18.032443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.033212 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.033213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:18.033308 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.033315 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:15:18.033351 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 15:15:18.033403 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":10,"thread_start_us":62,"threads_started":1}
I20260925 15:15:18.033710 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.029678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35093 (local address 127.8.43.130:38029)
0925 15:15:18.029895 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:18.029907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:18.029969 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:18.030141 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:18.030142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:18.030144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:18.030200 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:18.030201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.030996 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.030997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:18.032440 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:18.032443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.033215 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.033215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:18.033308 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.033314 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:15:18.033387 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:15:18.033416 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":34,"thread_start_us":91,"threads_started":1}
I20260925 15:15:18.033735 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.029608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:35669 (local address 127.8.43.129:33203)
0925 15:15:18.029895 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:18.029898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:18.029907 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:18.029940 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:18.029941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:18.029955 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:18.030063 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:18.030243 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.031015 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.032625 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.033077 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.033535 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.033547 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:18.033569 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 15:15:18.033659 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":71,"threads_started":1}
I20260925 15:15:18.033952 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b46358f2874b47a309c51c21d24be8" candidate_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98df7810e2ee46468799c563212402ab" is_pre_election: true
I20260925 15:15:18.034052 10246 raft_consensus.cc:2454] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d337488cfdb4b50ac7c89c7ad94ac22 in term 0.
I20260925 15:15:18.034231 10317 leader_election.cc:302] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d337488cfdb4b50ac7c89c7ad94ac22, 98df7810e2ee46468799c563212402ab; no voters:
I20260925 15:15:18.034446 10686 raft_consensus.cc:2781] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:18.034489 10686 raft_consensus.cc:482] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:18.034510 10686 raft_consensus.cc:3037] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:18.035369 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b46358f2874b47a309c51c21d24be8" candidate_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a4638187c148dcbcc0f0979135ce3c" is_pre_election: true
I20260925 15:15:18.035491 10526 raft_consensus.cc:2454] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d337488cfdb4b50ac7c89c7ad94ac22 in term 0.
I20260925 15:15:18.035459 10686 raft_consensus.cc:504] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.035583 10686 leader_election.cc:288] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [CANDIDATE]: Term 1 election: Requested vote from peers 98df7810e2ee46468799c563212402ab (127.8.43.129:33203), 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093)
I20260925 15:15:18.035777 10526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b46358f2874b47a309c51c21d24be8" candidate_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a4638187c148dcbcc0f0979135ce3c"
I20260925 15:15:18.035768 10246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b46358f2874b47a309c51c21d24be8" candidate_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98df7810e2ee46468799c563212402ab"
I20260925 15:15:18.035844 10526 raft_consensus.cc:3037] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:18.035840 10246 raft_consensus.cc:3037] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:18.036617 10246 raft_consensus.cc:2454] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d337488cfdb4b50ac7c89c7ad94ac22 in term 1.
I20260925 15:15:18.036617 10526 raft_consensus.cc:2454] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d337488cfdb4b50ac7c89c7ad94ac22 in term 1.
I20260925 15:15:18.036803 10317 leader_election.cc:302] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a4638187c148dcbcc0f0979135ce3c, 3d337488cfdb4b50ac7c89c7ad94ac22; no voters:
I20260925 15:15:18.036909 10686 raft_consensus.cc:2781] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:18.037050 10686 raft_consensus.cc:686] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 LEADER]: Becoming Leader. State: Replica: 3d337488cfdb4b50ac7c89c7ad94ac22, State: Running, Role: LEADER
I20260925 15:15:18.037135 10686 consensus_queue.cc:237] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } }
I20260925 15:15:18.037846 10089 catalog_manager.cc:5495] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130). New cstate: current_term: 1 leader_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d337488cfdb4b50ac7c89c7ad94ac22" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 } health_report { overall_health: UNKNOWN } } }
I20260925 15:15:18.045063 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.042201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.8.43.130:38059)
0925 15:15:18.042393 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:18.042395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:18.042403 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:18.042434 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:18.042435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:18.042452 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:18.042525 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:18.042664 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.043133 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.044209 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.044487 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.044549 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.044713 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:18.044814 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:18.044878 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:18.044942 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:15:18.044958 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260925 15:15:18.050173 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.047799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:42085 (local address 127.8.43.131:35093)
0925 15:15:18.047860 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:18.047862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:18.047875 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:15:18.047944 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:18.047951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:18.047961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:18.048019 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:18.048151 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.048625 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.049477 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.049670 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.049750 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.049923 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:18.049959 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:18.050020 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:18.050081 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:15:18.050100 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:15:18.050227 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.047616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35093 (local address 127.8.43.130:42085)
0925 15:15:18.047807 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:18.047834 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:18.047892 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:18.048023 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:18.048024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:18.048026 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:15:18.048085 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:18.048086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.048705 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.048706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:18.049383 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:18.049385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.049769 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.049770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:18.049827 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.049839 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:15:18.050021 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:15:18.050024 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:15:18.050076 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:15:18.050119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260925 15:15:18.053967 10526 raft_consensus.cc:1261] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Refusing update from remote peer 3d337488cfdb4b50ac7c89c7ad94ac22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:18.054116 10246 raft_consensus.cc:1261] T f2b46358f2874b47a309c51c21d24be8 P 98df7810e2ee46468799c563212402ab [term 1 FOLLOWER]: Refusing update from remote peer 3d337488cfdb4b50ac7c89c7ad94ac22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:18.054298 10702 consensus_queue.cc:1035] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a4638187c148dcbcc0f0979135ce3c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:18.054469 10686 consensus_queue.cc:1035] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98df7810e2ee46468799c563212402ab" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:18.062172 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.059643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:46247 (local address 127.8.43.130:38059)
0925 15:15:18.059681 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:18.059683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:18.059690 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:15:18.059712 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:18.059713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:18.059725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:18.059776 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:18.059962 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.060440 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.061489 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.061781 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.061849 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.062002 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:18.062025 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:18.062058 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:18.062114 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 15:15:18.062128 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:18.062228 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:18.059477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:38059 (local address 127.8.43.131:46247)
0925 15:15:18.059514 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:18.059525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:18.059566 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:18.059825 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:18.059826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:18.059827 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:15:18.059881 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:18.059884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.060529 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.060530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:18.061407 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:18.061410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:18.061861 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:18.061862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:18.061921 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:18.061928 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:15:18.062060 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:15:18.062062 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:15:18.062112 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 15:15:18.062156 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 15:15:18.062882 10651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:15:19.586457 10089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:19.586552 10089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:19.587363 10089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6cce67f67654782aa9b9816b8c0e3ad: Sending DeleteTablet for 3 replicas of tablet f2b46358f2874b47a309c51c21d24be8
I20260925 15:15:19.587662 10366 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b46358f2874b47a309c51c21d24be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:19 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:57008
I20260925 15:15:19.587687 10226 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b46358f2874b47a309c51c21d24be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:19 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:50538
I20260925 15:15:19.587761 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 10161
I20260925 15:15:19.587824 10506 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b46358f2874b47a309c51c21d24be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:19 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:42008
I20260925 15:15:19.589675 10735 tablet_replica.cc:330] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: stopping tablet replica
I20260925 15:15:19.589747 10735 raft_consensus.cc:2229] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:19.589787 10735 raft_consensus.cc:2258] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:19.589955 10733 tablet_replica.cc:330] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: stopping tablet replica
I20260925 15:15:19.590015 10733 raft_consensus.cc:2229] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:19.590030 10735 ts_tablet_manager.cc:1905] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:19.590159 10733 raft_consensus.cc:2258] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:19.590438 10733 ts_tablet_manager.cc:1905] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:19.591843 10735 ts_tablet_manager.cc:1918] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:15:19.591929 10735 log.cc:1196] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/f2b46358f2874b47a309c51c21d24be8
I20260925 15:15:19.592180 10735 ts_tablet_manager.cc:1939] T f2b46358f2874b47a309c51c21d24be8 P 01a4638187c148dcbcc0f0979135ce3c: Deleting consensus metadata
I20260925 15:15:19.592393 10733 ts_tablet_manager.cc:1918] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:15:19.592451 10733 log.cc:1196] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/f2b46358f2874b47a309c51c21d24be8
I20260925 15:15:19.592546 10073 catalog_manager.cc:4841] TS 01a4638187c148dcbcc0f0979135ce3c (127.8.43.131:35093): tablet f2b46358f2874b47a309c51c21d24be8 (table test-table [id=cb51dfcf72af40f49d8bfdd51f04c6c9]) successfully deleted
I20260925 15:15:19.592677 10733 ts_tablet_manager.cc:1939] T f2b46358f2874b47a309c51c21d24be8 P 3d337488cfdb4b50ac7c89c7ad94ac22: Deleting consensus metadata
I20260925 15:15:19.592972 10074 catalog_manager.cc:4841] TS 3d337488cfdb4b50ac7c89c7ad94ac22 (127.8.43.130:38059): tablet f2b46358f2874b47a309c51c21d24be8 (table test-table [id=cb51dfcf72af40f49d8bfdd51f04c6c9]) successfully deleted
W20260925 15:15:19.594578 10073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:33203 (error 108)
I20260925 15:15:19.594751 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 10301
I20260925 15:15:19.595649 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:19.595454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:33203 (local address 127.0.0.1:50548)
0925 15:15:19.595571 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:19.595616 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
W20260925 15:15:19.595796 10073 catalog_manager.cc:4584] TS 98df7810e2ee46468799c563212402ab (127.8.43.129:33203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2b46358f2874b47a309c51c21d24be8: Network error: Client connection negotiation failed: client connection to 127.8.43.129:33203: connect: Connection refused (error 111)
I20260925 15:15:19.601518 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 10441
I20260925 15:15:19.606865 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 10057
2026-09-25T15:15:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5835 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:19 dist-test-slave-c5d6 krb5kdc[10740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:19 dist-test-slave-c5d6 krb5kdc[10740](info): set up 2 sockets
Sep 25 15:15:19 dist-test-slave-c5d6 krb5kdc[10740](info): commencing operation
krb5kdc: starting...
W20260925 15:15:20.627064 8366 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:20 dist-test-slave-c5d6 krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:20.740022 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:36565
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:36565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:20.818511 10753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:20.818753 10753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:20.818787 10753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:20.819927 10753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:20.819996 10753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:20.820020 10753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:20.820041 10753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:20.820070 10753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:20.821514 10753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:36565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:36565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:20.821785 10753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:20.822006 10753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:20.826789 10761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:20.826907 10753 server_base.cc:1034] running on GCE node
W20260925 15:15:20.826758 10758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:20.826743 10759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:20.827284 10753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:20.827540 10753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:20.828681 10753 hybrid_clock.cc:648] HybridClock initialized: now 1790349320828667 us; error 28 us; skew 500 ppm
Sep 25 15:15:20 dist-test-slave-c5d6 krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:20.831095 10753 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:20.831529 10753 webserver.cc:460] Webserver started at http://127.8.43.190:37683/ using document root <none> and password file <none>
I20260925 15:15:20.831717 10753 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:20.831758 10753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:20.831843 10753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:20.832815 10753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "d04dd5fadfdc42fc80e22688f4924224"
format_stamp: "Formatted at 2026-09-25 15:15:20 on dist-test-slave-c5d6"
server_key: "0039e9cb0c7a4ec25a88d28a27968b1a"
server_key_iv: "eb3221ba399146938c2cbd8e9f836ebe"
server_key_version: "encryptionkey@0"
I20260925 15:15:20.833122 10753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "d04dd5fadfdc42fc80e22688f4924224"
format_stamp: "Formatted at 2026-09-25 15:15:20 on dist-test-slave-c5d6"
server_key: "0039e9cb0c7a4ec25a88d28a27968b1a"
server_key_iv: "eb3221ba399146938c2cbd8e9f836ebe"
server_key_version: "encryptionkey@0"
I20260925 15:15:20.834458 10753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:20.835110 10768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:20.835330 10753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:20.835435 10753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "d04dd5fadfdc42fc80e22688f4924224"
format_stamp: "Formatted at 2026-09-25 15:15:20 on dist-test-slave-c5d6"
server_key: "0039e9cb0c7a4ec25a88d28a27968b1a"
server_key_iv: "eb3221ba399146938c2cbd8e9f836ebe"
server_key_version: "encryptionkey@0"
I20260925 15:15:20.835497 10753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:20.862684 10753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:20.864622 10753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:20.864733 10753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:20.868635 10753 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:36565
I20260925 15:15:20.868687 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:36565 every 8 connection(s)
I20260925 15:15:20.869038 10753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:20.869920 10821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:20.872145 10821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224: Bootstrap starting.
I20260925 15:15:20.872663 10821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:20.872884 10821 log.cc:824] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:20.873540 10821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224: No bootstrap required, opened a new log
I20260925 15:15:20.874583 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 10753
I20260925 15:15:20.874656 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:20.874823 8366 external_mini_cluster.cc:1428] Setting key 2a13c3e1265064e870a2f8a00dbca130
I20260925 15:15:20.875216 10821 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04dd5fadfdc42fc80e22688f4924224" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36565 } }
I20260925 15:15:20.875416 10821 raft_consensus.cc:374] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:20.875479 10821 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04dd5fadfdc42fc80e22688f4924224, State: Initialized, Role: FOLLOWER
I20260925 15:15:20.875607 10821 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04dd5fadfdc42fc80e22688f4924224" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36565 } }
I20260925 15:15:20.875682 10821 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:20.875711 10821 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:20.875757 10821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:20.876686 10821 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04dd5fadfdc42fc80e22688f4924224" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36565 } }
I20260925 15:15:20.876788 10821 leader_election.cc:302] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04dd5fadfdc42fc80e22688f4924224; no voters:
I20260925 15:15:20.876935 10821 leader_election.cc:288] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:20.877008 10826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:20.877151 10826 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [term 1 LEADER]: Becoming Leader. State: Replica: d04dd5fadfdc42fc80e22688f4924224, State: Running, Role: LEADER
I20260925 15:15:20.877182 10821 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:20.877241 10826 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04dd5fadfdc42fc80e22688f4924224" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36565 } }
I20260925 15:15:20.877574 10826 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04dd5fadfdc42fc80e22688f4924224. Latest consensus state: current_term: 1 leader_uuid: "d04dd5fadfdc42fc80e22688f4924224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04dd5fadfdc42fc80e22688f4924224" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36565 } } }
I20260925 15:15:20.877660 10826 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:20.877583 10828 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d04dd5fadfdc42fc80e22688f4924224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04dd5fadfdc42fc80e22688f4924224" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36565 } } }
I20260925 15:15:20.877861 10828 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:20.877985 10834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:20.878443 10834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:20.880008 10834 catalog_manager.cc:1347] Generated new cluster ID: 39ae7c6643244a1faf9613bfd264109c
I20260925 15:15:20.880087 10834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:15:20 dist-test-slave-c5d6 krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:20.885679 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:20.875319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.8.43.190:36565)
0925 15:15:20.875500 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:20.875508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:20.875540 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:15:20.875649 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:20.875657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:20.875714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:20.875865 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:20.876477 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:20.877874 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:20.878983 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:20.879564 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:20.879664 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:20.882097 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:20.882100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:20.882101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:20.882112 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:20.883986 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:20.884383 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:20.884385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:20.884386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:20.884500 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:20.884706 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:20.884707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:20.884707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:20.884976 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:20.885116 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:20.885264 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:15:20.885385 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260925 15:15:20.893415 10834 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:20.893893 10834 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:20.897909 10834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d04dd5fadfdc42fc80e22688f4924224: Generated new TSK 0
I20260925 15:15:20.898072 10834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:20.926445 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36565
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:21.009033 10847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:21.009272 10847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:21.009301 10847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:21.010521 10847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:21.010577 10847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:21.010608 10847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:21.012171 10847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:21.012395 10847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:21.012672 10847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:21.018062 10853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:21.018075 10855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:21.018136 10847 server_base.cc:1034] running on GCE node
W20260925 15:15:21.018074 10852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:21.018575 10847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:21.018883 10847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:21.020040 10847 hybrid_clock.cc:648] HybridClock initialized: now 1790349321020023 us; error 30 us; skew 500 ppm
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:21.022792 10847 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:21.023350 10847 webserver.cc:460] Webserver started at http://127.8.43.129:41769/ using document root <none> and password file <none>
I20260925 15:15:21.023607 10847 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:21.023653 10847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:21.023799 10847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:21.024889 10847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "4acd801b128640b0aff2abbad13c607e"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "e88824ff462a858aced201a1b874a3b9"
server_key_iv: "7fac89d5c5324da23b744722a8cbf8a1"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.025246 10847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "4acd801b128640b0aff2abbad13c607e"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "e88824ff462a858aced201a1b874a3b9"
server_key_iv: "7fac89d5c5324da23b744722a8cbf8a1"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.026645 10847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:21.027488 10862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.027673 10847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:21.027756 10847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "4acd801b128640b0aff2abbad13c607e"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "e88824ff462a858aced201a1b874a3b9"
server_key_iv: "7fac89d5c5324da23b744722a8cbf8a1"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.027829 10847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:21.045830 10847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:21.047935 10847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:21.048094 10847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:21.048327 10847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:21.048651 10847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:21.048697 10847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.048739 10847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:21.048765 10847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.054466 10847 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:35703
I20260925 15:15:21.054514 10975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:35703 every 8 connection(s)
I20260925 15:15:21.054795 10847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:21.061561 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 10847
I20260925 15:15:21.061661 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:21.061849 8366 external_mini_cluster.cc:1428] Setting key c2a20ed56c00afa0e4f82b8b925e8993
I20260925 15:15:21.064050 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:21.055496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:45327 (local address 127.8.43.190:36565)
0925 15:15:21.055570 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:21.055572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:21.055716 (+ 144us) server_negotiation.cc:407] Connection header received
0925 15:15:21.056161 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:21.056162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:21.056172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:21.056221 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:21.057085 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.057650 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.058630 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.058818 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.058876 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:21.060966 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:21.060967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:21.060968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:21.060976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:21.062623 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:21.063022 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:21.063024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:21.063024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:21.063079 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:21.063389 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:21.063390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:21.063390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:21.063503 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:21.063577 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:21.063872 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 15:15:21.063959 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:15:21.064231 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:21.055444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36565 (local address 127.8.43.129:45327)
0925 15:15:21.055608 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:21.055642 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:21.056086 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:21.056273 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:21.056276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:21.056797 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:21.056986 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:21.056990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.057729 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.057730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:21.058552 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:21.058553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.058890 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.058890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:21.058993 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:21.059530 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:21.059546 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:21.060857 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:21.062709 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:21.062715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:21.062718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:21.062945 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:21.063155 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:21.063156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:21.063157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:21.063321 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:21.063571 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:21.063576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:21.063723 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 15:15:21.063821 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:15:21.063995 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260925 15:15:21.064679 10976 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36565
I20260925 15:15:21.064766 10976 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:21.064941 10976 heartbeater.cc:511] Master 127.8.43.190:36565 requested a full tablet report, sending...
I20260925 15:15:21.065451 10785 ts_manager.cc:194] Registered new tserver with Master: 4acd801b128640b0aff2abbad13c607e (127.8.43.129:35703)
I20260925 15:15:21.066057 10785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:45327
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:21.086108 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36565
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:21.165069 10981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:21.165293 10981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:21.165329 10981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:21.166493 10981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:21.166556 10981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:21.166600 10981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:21.167994 10981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:21.168239 10981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:21.168455 10981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:21.173404 10986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:21.173398 10987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:21.173398 10989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:21.173870 10981 server_base.cc:1034] running on GCE node
I20260925 15:15:21.174139 10981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:21.174371 10981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:21.175530 10981 hybrid_clock.cc:648] HybridClock initialized: now 1790349321175510 us; error 27 us; skew 500 ppm
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:21.178079 10981 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:21.178432 10981 webserver.cc:460] Webserver started at http://127.8.43.130:45941/ using document root <none> and password file <none>
I20260925 15:15:21.178609 10981 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:21.178649 10981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:21.178743 10981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:21.179833 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "07e43fe5d5424d31bfa0925cc88548a5"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "d409cd9fa2da62ea80a4a87506c5122a"
server_key_iv: "da947f7878e7f7b6911611d20b581a22"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.180183 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "07e43fe5d5424d31bfa0925cc88548a5"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "d409cd9fa2da62ea80a4a87506c5122a"
server_key_iv: "da947f7878e7f7b6911611d20b581a22"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.181383 10981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:21.182081 10996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.182242 10981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:21.182339 10981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "07e43fe5d5424d31bfa0925cc88548a5"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "d409cd9fa2da62ea80a4a87506c5122a"
server_key_iv: "da947f7878e7f7b6911611d20b581a22"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.182415 10981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:21.191218 10981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:21.192906 10981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:21.193056 10981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:21.193289 10981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:21.193569 10981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:21.193621 10981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.193660 10981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:21.193686 10981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.200147 10981 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:34683
I20260925 15:15:21.200263 11109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:34683 every 8 connection(s)
I20260925 15:15:21.200546 10981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:21.208949 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:21.201184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:60637 (local address 127.8.43.190:36565)
0925 15:15:21.201225 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:21.201227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:21.201435 (+ 208us) server_negotiation.cc:407] Connection header received
0925 15:15:21.201941 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:21.201942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:21.201952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:21.202001 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:21.202847 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.203387 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.204396 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.204592 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.204645 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:21.206670 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:21.206671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:21.206672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:21.206678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:21.207890 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:21.208232 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:21.208232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:21.208233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:21.208279 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:21.208476 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:21.208476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:21.208476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:21.208571 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:21.208618 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:21.208829 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 15:15:21.208889 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:21.209095 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:21.201161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36565 (local address 127.8.43.130:60637)
0925 15:15:21.201297 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:21.201328 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:21.201848 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:21.202054 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:21.202058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:21.202606 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:21.202777 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:21.202781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.203457 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.203458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:21.204329 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:21.204331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.204657 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.204658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:21.204746 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:21.205240 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:21.205247 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:21.206588 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:21.207959 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:21.207964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:21.207966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:21.208168 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:21.208337 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:21.208337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:21.208337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:21.208425 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:21.208623 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:21.208627 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:21.208726 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 15:15:21.208804 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:15:21.208909 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260925 15:15:21.209560 11110 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36565
I20260925 15:15:21.209672 11110 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:21.209833 11110 heartbeater.cc:511] Master 127.8.43.190:36565 requested a full tablet report, sending...
I20260925 15:15:21.210176 10785 ts_manager.cc:194] Registered new tserver with Master: 07e43fe5d5424d31bfa0925cc88548a5 (127.8.43.130:34683)
I20260925 15:15:21.210417 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 10981
I20260925 15:15:21.210517 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:21.210659 10785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:60637
I20260925 15:15:21.210703 8366 external_mini_cluster.cc:1428] Setting key fe23e7b588f048c0aa8e825f2cef3800
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:21.232334 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36565
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:21.312641 11115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:21.312841 11115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:21.312865 11115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:21.314298 11115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:21.314407 11115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:21.314495 11115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:21.316154 11115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:21.316421 11115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:21.316682 11115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:21.321761 11120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:21.321808 11121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:21.321802 11123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:21.322085 11115 server_base.cc:1034] running on GCE node
I20260925 15:15:21.322312 11115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:21.322589 11115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:21.323786 11115 hybrid_clock.cc:648] HybridClock initialized: now 1790349321323784 us; error 45 us; skew 500 ppm
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:21.326372 11115 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:21.326813 11115 webserver.cc:460] Webserver started at http://127.8.43.131:38373/ using document root <none> and password file <none>
I20260925 15:15:21.327008 11115 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:21.327049 11115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:21.327139 11115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:21.328266 11115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "38c54701ef634ad7a62e527b3862d8c4"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "7147b495f3c5fb996603dced2dce5ef1"
server_key_iv: "6f2b55d22b0a5a18c3f0c1ad35407dd4"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.328559 11115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "38c54701ef634ad7a62e527b3862d8c4"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "7147b495f3c5fb996603dced2dce5ef1"
server_key_iv: "6f2b55d22b0a5a18c3f0c1ad35407dd4"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.329805 11115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:15:21.330524 11130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.330686 11115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:21.330767 11115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "38c54701ef634ad7a62e527b3862d8c4"
format_stamp: "Formatted at 2026-09-25 15:15:21 on dist-test-slave-c5d6"
server_key: "7147b495f3c5fb996603dced2dce5ef1"
server_key_iv: "6f2b55d22b0a5a18c3f0c1ad35407dd4"
server_key_version: "encryptionkey@0"
I20260925 15:15:21.330818 11115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:21.344441 11115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:21.346127 11115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:21.346228 11115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:21.346427 11115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:21.346666 11115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:21.346693 11115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.346714 11115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:21.346726 11115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:21.352207 11115 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:34163
I20260925 15:15:21.352280 11243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:34163 every 8 connection(s)
I20260925 15:15:21.352566 11115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:21.356585 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 11115
I20260925 15:15:21.356665 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:21.356843 8366 external_mini_cluster.cc:1428] Setting key 5b6d9ebfd9efd1b34c29f6c707e474db
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:21.361397 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:21.353229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:38269 (local address 127.8.43.190:36565)
0925 15:15:21.353292 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:21.353294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:21.353514 (+ 220us) server_negotiation.cc:407] Connection header received
0925 15:15:21.354023 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:21.354024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:21.354037 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:21.354086 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:21.354948 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.355514 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.356486 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.356728 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.356794 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:21.358957 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:21.358959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:21.358959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:21.358966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:21.360118 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:21.360497 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:21.360498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:21.360499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:21.360556 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:21.360788 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:21.360789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:21.360790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:21.360898 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:21.360981 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:21.361235 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 15:15:21.361314 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:15:21.361596 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:21.353208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36565 (local address 127.8.43.131:38269)
0925 15:15:21.353347 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:21.353410 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:21.353916 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:21.354151 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:21.354155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:21.354674 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:21.354842 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:21.354846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.355599 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.355600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:21.356378 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:21.356380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:21.356801 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:21.356801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:21.356945 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:21.357418 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:21.357449 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:21.358864 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:21.360195 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:21.360202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:21.360204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:21.360409 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:21.360620 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:21.360620 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:21.360621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:21.360723 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:21.360971 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:21.360976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:21.361107 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 15:15:21.361201 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:15:21.361338 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260925 15:15:21.362004 11244 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36565
I20260925 15:15:21.362087 11244 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:21.362210 11244 heartbeater.cc:511] Master 127.8.43.190:36565 requested a full tablet report, sending...
I20260925 15:15:21.362531 10785 ts_manager.cc:194] Registered new tserver with Master: 38c54701ef634ad7a62e527b3862d8c4 (127.8.43.131:34163)
I20260925 15:15:21.362938 10785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:38269
I20260925 15:15:21.368669 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:15:21.379741 10825 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:21.378746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.8.43.190:36565)
0925 15:15:21.378801 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:21.378803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:21.378912 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:15:21.379014 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:21.379016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:21.379028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:21.379096 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:21.379691 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36142: BlockingRecv error: recv got EOF from 127.0.0.1:36142 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:15:21.380697 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 10847
I20260925 15:15:21.385012 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 10981
I20260925 15:15:21.389094 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 11115
I20260925 15:15:21.393196 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 10753
2026-09-25T15:15:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1785 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[11256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[11256](info): set up 2 sockets
Sep 25 15:15:21 dist-test-slave-c5d6 krb5kdc[11256](info): commencing operation
krb5kdc: starting...
W20260925 15:15:22.419397 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:22 dist-test-slave-c5d6 krb5kdc[11256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:22.536178 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:37513
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:22.616875 11269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:22.617123 11269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:22.617148 11269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:22.618261 11269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:22.618295 11269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:22.618309 11269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:22.618322 11269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:22.618335 11269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:22.619768 11269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:37513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:37513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:22.619969 11269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:22.620221 11269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:22.625164 11274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:22.625164 11277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:22.625237 11275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:22.625406 11269 server_base.cc:1034] running on GCE node
I20260925 15:15:22.625717 11269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:22.626037 11269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:22.627259 11269 hybrid_clock.cc:648] HybridClock initialized: now 1790349322627196 us; error 43 us; skew 500 ppm
Sep 25 15:15:22 dist-test-slave-c5d6 krb5kdc[11256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349322, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:22.629874 11269 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:22.630379 11269 webserver.cc:460] Webserver started at http://127.8.43.190:45621/ using document root <none> and password file <none>
I20260925 15:15:22.630615 11269 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:22.630685 11269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:22.630841 11269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:22.632010 11269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "f68ae478120149b3a58f42493afabb7d"
format_stamp: "Formatted at 2026-09-25 15:15:22 on dist-test-slave-c5d6"
server_key: "04168c4ca667dace737b12da7f8fbe23"
server_key_iv: "6c31d57715e009955083ddbbf9b03d42"
server_key_version: "encryptionkey@0"
I20260925 15:15:22.632370 11269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "f68ae478120149b3a58f42493afabb7d"
format_stamp: "Formatted at 2026-09-25 15:15:22 on dist-test-slave-c5d6"
server_key: "04168c4ca667dace737b12da7f8fbe23"
server_key_iv: "6c31d57715e009955083ddbbf9b03d42"
server_key_version: "encryptionkey@0"
I20260925 15:15:22.633714 11269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:22.634438 11284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:22.634644 11269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:22.634759 11269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "f68ae478120149b3a58f42493afabb7d"
format_stamp: "Formatted at 2026-09-25 15:15:22 on dist-test-slave-c5d6"
server_key: "04168c4ca667dace737b12da7f8fbe23"
server_key_iv: "6c31d57715e009955083ddbbf9b03d42"
server_key_version: "encryptionkey@0"
I20260925 15:15:22.634838 11269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:22.643896 11269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:22.645828 11269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:22.645999 11269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:22.649950 11269 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:37513
I20260925 15:15:22.649992 11336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:37513 every 8 connection(s)
I20260925 15:15:22.650651 11269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:22.650789 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 11269
I20260925 15:15:22.650892 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:22.651106 8366 external_mini_cluster.cc:1428] Setting key 2e3ca6668c4df0e4595138f055a59409
I20260925 15:15:22.652056 11337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:22.654881 11337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d: Bootstrap starting.
I20260925 15:15:22.655601 11337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:22.655884 11337 log.cc:824] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:22.656595 11337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d: No bootstrap required, opened a new log
Sep 25 15:15:22 dist-test-slave-c5d6 krb5kdc[11256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:22.658978 11337 raft_consensus.cc:348] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68ae478120149b3a58f42493afabb7d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37513 } }
I20260925 15:15:22.659165 11337 raft_consensus.cc:374] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:22.659205 11337 raft_consensus.cc:729] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68ae478120149b3a58f42493afabb7d, State: Initialized, Role: FOLLOWER
I20260925 15:15:22.659401 11337 consensus_queue.cc:260] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68ae478120149b3a58f42493afabb7d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37513 } }
I20260925 15:15:22.659482 11337 raft_consensus.cc:388] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:22.659504 11337 raft_consensus.cc:482] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:22.659581 11337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:22.660238 11337 raft_consensus.cc:504] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68ae478120149b3a58f42493afabb7d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37513 } }
I20260925 15:15:22.660324 11337 leader_election.cc:302] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f68ae478120149b3a58f42493afabb7d; no voters:
I20260925 15:15:22.660544 11337 leader_election.cc:288] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:22.660604 11342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:22.660750 11342 raft_consensus.cc:686] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [term 1 LEADER]: Becoming Leader. State: Replica: f68ae478120149b3a58f42493afabb7d, State: Running, Role: LEADER
I20260925 15:15:22.660817 11337 sys_catalog.cc:564] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:22.660873 11342 consensus_queue.cc:237] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68ae478120149b3a58f42493afabb7d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37513 } }
I20260925 15:15:22.661216 11342 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f68ae478120149b3a58f42493afabb7d. Latest consensus state: current_term: 1 leader_uuid: "f68ae478120149b3a58f42493afabb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68ae478120149b3a58f42493afabb7d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37513 } } }
I20260925 15:15:22.661310 11342 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:22.661226 11344 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f68ae478120149b3a58f42493afabb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68ae478120149b3a58f42493afabb7d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37513 } } }
I20260925 15:15:22.661484 11344 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:22.662179 11349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:22.663015 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:22.651624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.8.43.190:37513)
0925 15:15:22.651810 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:22.651820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:22.651852 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:15:22.652436 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:22.652444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:22.652504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:22.652652 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:22.653245 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:22.654591 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:22.655497 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:22.656132 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:22.656258 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:22.658825 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:22.658829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:22.658829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:22.658841 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:22.660864 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:22.661293 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:22.661296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:22.661296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:22.661403 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:22.661635 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:22.661636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:22.661637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:22.662008 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:22.662601 (+ 593us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:22.662688 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:15:22.662831 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260925 15:15:22.663149 11349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:22.663910 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:15:22.664711 11349 catalog_manager.cc:1347] Generated new cluster ID: ff2a612c6ef44d1e9efe128750e40dc5
I20260925 15:15:22.664754 11349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:22.681592 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:22.678093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.8.43.190:37513)
0925 15:15:22.678139 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:22.678141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:22.678281 (+ 140us) server_negotiation.cc:407] Connection header received
0925 15:15:22.678354 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:22.678355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:22.678365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:22.678437 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:22.678819 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:22.679364 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:22.680064 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:22.680285 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:22.680368 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:22.681069 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:22.681070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:22.681071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:22.681307 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:22.681322 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:22.681328 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:22.681391 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:22.681454 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 15:15:22.681475 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:15:22.691993 11349 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:22.692701 11349 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:22.696583 11349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f68ae478120149b3a58f42493afabb7d: Generated new TSK 0
I20260925 15:15:22.696755 11349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:15:22.703367 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 11269
2026-09-25T15:15:22Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1308 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:22 dist-test-slave-c5d6 krb5kdc[11370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:22 dist-test-slave-c5d6 krb5kdc[11370](info): set up 2 sockets
Sep 25 15:15:22 dist-test-slave-c5d6 krb5kdc[11370](info): commencing operation
krb5kdc: starting...
W20260925 15:15:23.731184 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:23 dist-test-slave-c5d6 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:23.848065 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:35273
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:35273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:23.932082 11383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:23.932305 11383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:23.932343 11383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:23.933504 11383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:23.933566 11383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:23.933591 11383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:23.933621 11383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:23.933646 11383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:23.935073 11383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:35273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:35273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:23.935361 11383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:23.935647 11383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:23.940440 11388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:23.940529 11389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:23.940605 11383 server_base.cc:1034] running on GCE node
W20260925 15:15:23.940485 11391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:23.940944 11383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:23.941294 11383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:23.942474 11383 hybrid_clock.cc:648] HybridClock initialized: now 1790349323942461 us; error 46 us; skew 500 ppm
Sep 25 15:15:23 dist-test-slave-c5d6 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349323, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:23.944962 11383 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:23.945374 11383 webserver.cc:460] Webserver started at http://127.8.43.190:39915/ using document root <none> and password file <none>
I20260925 15:15:23.945624 11383 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:23.945693 11383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:23.945837 11383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:23.946918 11383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "5a7172aa79d7499792d2cd42ea367b30"
format_stamp: "Formatted at 2026-09-25 15:15:23 on dist-test-slave-c5d6"
server_key: "00b92b52407f1ad5aedc37661f563fe5"
server_key_iv: "80083119cde778d22dac709c6989f458"
server_key_version: "encryptionkey@0"
I20260925 15:15:23.947326 11383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "5a7172aa79d7499792d2cd42ea367b30"
format_stamp: "Formatted at 2026-09-25 15:15:23 on dist-test-slave-c5d6"
server_key: "00b92b52407f1ad5aedc37661f563fe5"
server_key_iv: "80083119cde778d22dac709c6989f458"
server_key_version: "encryptionkey@0"
I20260925 15:15:23.948632 11383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:23.949303 11398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:23.949501 11383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:23.949597 11383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "5a7172aa79d7499792d2cd42ea367b30"
format_stamp: "Formatted at 2026-09-25 15:15:23 on dist-test-slave-c5d6"
server_key: "00b92b52407f1ad5aedc37661f563fe5"
server_key_iv: "80083119cde778d22dac709c6989f458"
server_key_version: "encryptionkey@0"
I20260925 15:15:23.949680 11383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:23.966886 11383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:23.968935 11383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:23.969105 11383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:23.972772 11383 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:35273
I20260925 15:15:23.972808 11450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:35273 every 8 connection(s)
I20260925 15:15:23.973181 11383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:23.974228 11451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:23.976526 11451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30: Bootstrap starting.
I20260925 15:15:23.977041 11451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:23.977288 11451 log.cc:824] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:23.978058 11451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30: No bootstrap required, opened a new log
I20260925 15:15:23.979718 11451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7172aa79d7499792d2cd42ea367b30" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35273 } }
I20260925 15:15:23.979859 11451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:23.979882 11451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a7172aa79d7499792d2cd42ea367b30, State: Initialized, Role: FOLLOWER
I20260925 15:15:23.980019 11451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7172aa79d7499792d2cd42ea367b30" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35273 } }
I20260925 15:15:23.980099 11451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:23.980122 11451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:23.980150 11451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:23.980911 11451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7172aa79d7499792d2cd42ea367b30" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35273 } }
I20260925 15:15:23.980994 11451 leader_election.cc:302] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a7172aa79d7499792d2cd42ea367b30; no voters:
I20260925 15:15:23.981145 11451 leader_election.cc:288] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:23.981169 11454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:23.981285 11454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [term 1 LEADER]: Becoming Leader. State: Replica: 5a7172aa79d7499792d2cd42ea367b30, State: Running, Role: LEADER
I20260925 15:15:23.981398 11451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:23.981427 11454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7172aa79d7499792d2cd42ea367b30" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35273 } }
I20260925 15:15:23.981849 11455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a7172aa79d7499792d2cd42ea367b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7172aa79d7499792d2cd42ea367b30" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35273 } } }
I20260925 15:15:23.981884 11456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a7172aa79d7499792d2cd42ea367b30. Latest consensus state: current_term: 1 leader_uuid: "5a7172aa79d7499792d2cd42ea367b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7172aa79d7499792d2cd42ea367b30" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35273 } } }
I20260925 15:15:23.982016 11455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:23.982030 11456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:23.982537 11463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:23.982954 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 11383
I20260925 15:15:23.983062 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:23.983126 11463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:23.983281 8366 external_mini_cluster.cc:1428] Setting key 2a9301786a5530ff84f61d4c357c15cf
I20260925 15:15:23.984912 11463 catalog_manager.cc:1347] Generated new cluster ID: 40342f6ca6374c8ca24d5878bfda08cf
I20260925 15:15:23.984969 11463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:15:23 dist-test-slave-c5d6 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:23.995201 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:23.983910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.8.43.190:35273)
0925 15:15:23.984063 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:23.984072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:23.984108 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:15:23.984198 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:23.984203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:23.984247 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:23.984414 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:23.985146 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:23.986564 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:23.987707 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:23.988317 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:23.988426 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:23.991373 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:23.991377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:23.991378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:23.991391 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:23.993463 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:23.993862 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:23.993865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:23.993865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:23.993977 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:23.994196 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:23.994197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:23.994197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:23.994521 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:23.994683 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:23.994798 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:15:23.994951 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260925 15:15:24.000160 11463 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:24.000728 11463 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:24.003484 11463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a7172aa79d7499792d2cd42ea367b30: Generated new TSK 0
I20260925 15:15:24.003674 11463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:24.028311 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35273
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:24.107139 11477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:24.107347 11477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:24.107368 11477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:24.108466 11477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:24.108500 11477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:24.108525 11477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:24.109858 11477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:24.110059 11477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:24.110249 11477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:24.115161 11483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:24.115183 11485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:24.115168 11482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:24.115406 11477 server_base.cc:1034] running on GCE node
I20260925 15:15:24.115743 11477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:24.116007 11477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:24.117179 11477 hybrid_clock.cc:648] HybridClock initialized: now 1790349324117157 us; error 33 us; skew 500 ppm
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:24.119560 11477 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:24.119992 11477 webserver.cc:460] Webserver started at http://127.8.43.129:39049/ using document root <none> and password file <none>
I20260925 15:15:24.120220 11477 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:24.120285 11477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:24.120440 11477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:24.121522 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "6d7296c31f214a80888b40b654328e2b"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "f3444b7dfe246338848f47014860cc02"
server_key_iv: "84d82ceb691958ea2c6ba4b94f1c8eb9"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.121878 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "6d7296c31f214a80888b40b654328e2b"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "f3444b7dfe246338848f47014860cc02"
server_key_iv: "84d82ceb691958ea2c6ba4b94f1c8eb9"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.123270 11477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 15:15:24.124120 11492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.124328 11477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:24.124426 11477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "6d7296c31f214a80888b40b654328e2b"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "f3444b7dfe246338848f47014860cc02"
server_key_iv: "84d82ceb691958ea2c6ba4b94f1c8eb9"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.124497 11477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:24.131989 11477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:24.133754 11477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:24.133915 11477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:24.134131 11477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:24.134423 11477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:24.134476 11477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.134516 11477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:24.134542 11477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.140589 11477 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:40317
I20260925 15:15:24.140640 11605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:40317 every 8 connection(s)
I20260925 15:15:24.140924 11477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:24.142746 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 11477
I20260925 15:15:24.142823 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:24.143018 8366 external_mini_cluster.cc:1428] Setting key d96e6157d40e4912aea56d2b624ae628
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
I20260925 15:15:24.150878 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:24.141616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:48101 (local address 127.8.43.190:35273)
0925 15:15:24.141674 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:24.141677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:24.141898 (+ 221us) server_negotiation.cc:407] Connection header received
0925 15:15:24.142336 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:24.142338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:24.142348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:24.142398 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:24.143565 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.144153 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.145199 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.145417 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.145488 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:24.148081 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:24.148082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:24.148083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:24.148089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:24.149492 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:24.149966 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:24.149967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:24.149967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:24.150015 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:24.150289 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:24.150290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:24.150290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:24.150383 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:24.150430 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:24.150747 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 15:15:24.150813 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:24.151150 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:24.141603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35273 (local address 127.8.43.129:48101)
0925 15:15:24.141765 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:24.141797 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:24.142257 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:24.142455 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:24.142458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:24.143117 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:24.143461 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:24.143467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.144258 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.144259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:24.145110 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:24.145112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.145504 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.145505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:24.145705 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:24.146195 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:24.146211 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:24.147993 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:24.149577 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:24.149582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:24.149585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:24.149879 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:24.150082 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:24.150083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:24.150083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:24.150218 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:24.150447 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:24.150454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:24.150615 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 15:15:24.150728 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:15:24.150884 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260925 15:15:24.151691 11606 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35273
I20260925 15:15:24.151779 11606 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:24.151926 11606 heartbeater.cc:511] Master 127.8.43.190:35273 requested a full tablet report, sending...
I20260925 15:15:24.152561 11415 ts_manager.cc:194] Registered new tserver with Master: 6d7296c31f214a80888b40b654328e2b (127.8.43.129:40317)
I20260925 15:15:24.153196 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:48101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:24.166119 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35273
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:24.246469 11611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:24.246649 11611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:24.246670 11611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:24.247748 11611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:24.247784 11611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:24.247809 11611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:24.249136 11611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:24.249302 11611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:24.249470 11611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:15:24.254364 11611 server_base.cc:1034] running on GCE node
W20260925 15:15:24.254352 11619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:24.254328 11617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:24.254333 11616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:24.254716 11611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:24.254958 11611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:24.256115 11611 hybrid_clock.cc:648] HybridClock initialized: now 1790349324256099 us; error 27 us; skew 500 ppm
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:24.258755 11611 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:24.259389 11611 webserver.cc:460] Webserver started at http://127.8.43.130:45649/ using document root <none> and password file <none>
I20260925 15:15:24.259682 11611 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:24.259764 11611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:24.259965 11611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:24.261147 11611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "7aadf2c585ad40c19818b7181dff6731"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "408f20c3b19cea4d9f8af1b6d1ba3c69"
server_key_iv: "f3870890ac0907f9d45a5e6c7ea727e7"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.261575 11611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "7aadf2c585ad40c19818b7181dff6731"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "408f20c3b19cea4d9f8af1b6d1ba3c69"
server_key_iv: "f3870890ac0907f9d45a5e6c7ea727e7"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.263170 11611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 15:15:24.263978 11626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.264145 11611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:24.264233 11611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "7aadf2c585ad40c19818b7181dff6731"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "408f20c3b19cea4d9f8af1b6d1ba3c69"
server_key_iv: "f3870890ac0907f9d45a5e6c7ea727e7"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.264312 11611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:24.277035 11611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:24.279407 11611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:24.279601 11611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:24.279870 11611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:24.280166 11611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:24.280218 11611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.280251 11611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:24.280293 11611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.286378 11611 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:42013
I20260925 15:15:24.286428 11739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:42013 every 8 connection(s)
I20260925 15:15:24.286738 11611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:24.290851 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 11611
I20260925 15:15:24.290972 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:24.291172 8366 external_mini_cluster.cc:1428] Setting key 6aa50ae99bb6c067b5a0db9cfb901643
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:24.297933 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:24.287575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:58241 (local address 127.8.43.190:35273)
0925 15:15:24.287656 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:24.287658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:24.287790 (+ 132us) server_negotiation.cc:407] Connection header received
0925 15:15:24.288360 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:24.288362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:24.288373 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:24.288432 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:24.289464 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.289949 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.291332 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.291551 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.291614 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:24.295086 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:24.295088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:24.295088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:24.295097 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:24.296546 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:24.296964 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:24.296965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:24.296966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:24.297027 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:24.297287 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:24.297287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:24.297288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:24.297407 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:24.297484 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:24.297754 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 15:15:24.297831 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:15:24.298117 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:24.287461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35273 (local address 127.8.43.130:58241)
0925 15:15:24.287661 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:24.287701 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:24.288235 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:24.288519 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:24.288523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:24.289152 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:24.289359 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:24.289364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.290035 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.290036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:24.291243 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:24.291247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.291645 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.291646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:24.291801 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:24.293478 (+ 1677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:24.293487 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:24.294987 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:24.296644 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:24.296650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:24.296652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:24.296882 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:24.297094 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:24.297094 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:24.297095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:24.297212 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:24.297475 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:24.297480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:24.297621 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:15:24.297723 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:15:24.297877 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260925 15:15:24.298513 11740 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35273
I20260925 15:15:24.298607 11740 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:24.298786 11740 heartbeater.cc:511] Master 127.8.43.190:35273 requested a full tablet report, sending...
I20260925 15:15:24.299161 11415 ts_manager.cc:194] Registered new tserver with Master: 7aadf2c585ad40c19818b7181dff6731 (127.8.43.130:42013)
I20260925 15:15:24.299559 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:58241
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:24.321975 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35273
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:24.410981 11745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:24.411243 11745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:24.411324 11745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:24.412663 11745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:24.412712 11745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:24.412742 11745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:24.414369 11745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:24.414680 11745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:24.414963 11745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:24.420224 11751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:24.420202 11753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:24.420212 11750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:24.420598 11745 server_base.cc:1034] running on GCE node
I20260925 15:15:24.420864 11745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:24.421119 11745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:24.422292 11745 hybrid_clock.cc:648] HybridClock initialized: now 1790349324422281 us; error 26 us; skew 500 ppm
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:24.425237 11745 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:24.425736 11745 webserver.cc:460] Webserver started at http://127.8.43.131:35499/ using document root <none> and password file <none>
I20260925 15:15:24.425998 11745 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:24.426074 11745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:24.426241 11745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:24.427477 11745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "e198eae438bd415c94f19331bc60480e"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "d80aec0643830ed709fe2c6a68516735"
server_key_iv: "a26639ca0924ce59b898e12637ab65b2"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.427855 11745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "e198eae438bd415c94f19331bc60480e"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "d80aec0643830ed709fe2c6a68516735"
server_key_iv: "a26639ca0924ce59b898e12637ab65b2"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.429212 11745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:24.430017 11760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.430202 11745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:24.430308 11745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "e198eae438bd415c94f19331bc60480e"
format_stamp: "Formatted at 2026-09-25 15:15:24 on dist-test-slave-c5d6"
server_key: "d80aec0643830ed709fe2c6a68516735"
server_key_iv: "a26639ca0924ce59b898e12637ab65b2"
server_key_version: "encryptionkey@0"
I20260925 15:15:24.430393 11745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:24.446151 11745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:24.448101 11745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:24.448264 11745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:24.448499 11745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:24.448843 11745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:24.448901 11745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.448942 11745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:24.448974 11745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:24.455861 11745 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:37321
I20260925 15:15:24.455928 11873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:37321 every 8 connection(s)
I20260925 15:15:24.456264 11745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:24.457016 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 11745
I20260925 15:15:24.457139 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:24.457376 8366 external_mini_cluster.cc:1428] Setting key f220c62c69a924fd23d40640427b4d1f
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:24.465713 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:24.457006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:40151 (local address 127.8.43.190:35273)
0925 15:15:24.457051 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:24.457054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:24.457323 (+ 269us) server_negotiation.cc:407] Connection header received
0925 15:15:24.458028 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:24.458029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:24.458042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:24.458102 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:24.459106 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.459618 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.460676 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.460917 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.460994 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:24.463078 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:24.463079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:24.463079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:24.463087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:24.464558 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:24.464961 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:24.464962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:24.464962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:24.465012 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:24.465232 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:24.465233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:24.465233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:24.465324 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:24.465370 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:24.465598 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:15:24.465653 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:24.465962 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:24.456988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35273 (local address 127.8.43.131:40151)
0925 15:15:24.457164 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:24.457213 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:24.457914 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:24.458168 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:24.458172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:24.458817 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:24.459012 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:24.459016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.459708 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.459709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:24.460578 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:24.460580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.461003 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.461004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:24.461106 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:24.461557 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:24.461565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:24.462975 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:24.464642 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:24.464647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:24.464649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:24.464868 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:24.465072 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:24.465072 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:24.465073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:24.465179 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:24.465376 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:24.465380 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:24.465492 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 15:15:24.465573 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:15:24.465685 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260925 15:15:24.466470 11874 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35273
I20260925 15:15:24.466578 11874 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:24.466750 11874 heartbeater.cc:511] Master 127.8.43.190:35273 requested a full tablet report, sending...
I20260925 15:15:24.467063 11415 ts_manager.cc:194] Registered new tserver with Master: e198eae438bd415c94f19331bc60480e (127.8.43.131:37321)
I20260925 15:15:24.467476 11415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:40151
I20260925 15:15:24.469040 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349324, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349324, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:24.492734 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:24.485684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.8.43.190:35273)
0925 15:15:24.485740 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:24.485742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:24.485755 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:15:24.485802 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:24.485803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:24.485811 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:24.485869 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:24.486505 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.486988 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.487795 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:24.487997 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:24.488054 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:24.490285 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:24.490286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:24.490286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:24.490293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:24.491715 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:24.492026 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:24.492027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:24.492027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:24.492080 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:24.492267 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:24.492267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:24.492268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:24.492421 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:24.492519 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:24.492590 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:15:24.492646 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 15:15:24.492995 11415 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58152
I20260925 15:15:24.493417 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 11477
I20260925 15:15:24.497911 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 11611
I20260925 15:15:24.502009 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 11745
I20260925 15:15:24.506577 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 11383
2026-09-25T15:15:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1805 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11886](info): set up 2 sockets
Sep 25 15:15:24 dist-test-slave-c5d6 krb5kdc[11886](info): commencing operation
krb5kdc: starting...
W20260925 15:15:25.547741 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:25 dist-test-slave-c5d6 krb5kdc[11886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:25.667095 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:45527
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:45527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:25.749387 11899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:25.749629 11899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:25.749655 11899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:25.751003 11899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:25.751048 11899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:25.751065 11899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:25.751078 11899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:25.751092 11899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:25.752882 11899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:45527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:25.753158 11899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:25.753436 11899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:25.758720 11904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:25.758716 11905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:25.758713 11907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:25.758795 11899 server_base.cc:1034] running on GCE node
I20260925 15:15:25.759115 11899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:25.759445 11899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:25.760617 11899 hybrid_clock.cc:648] HybridClock initialized: now 1790349325760594 us; error 37 us; skew 500 ppm
Sep 25 15:15:25 dist-test-slave-c5d6 krb5kdc[11886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:25.763093 11899 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:25.763537 11899 webserver.cc:460] Webserver started at http://127.8.43.190:43671/ using document root <none> and password file <none>
I20260925 15:15:25.763761 11899 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:25.763808 11899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:25.763934 11899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:25.765003 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "be6f43b4cc884646a2eaf0ea5523580b"
format_stamp: "Formatted at 2026-09-25 15:15:25 on dist-test-slave-c5d6"
server_key: "0e38f2d3c7e05276c64083187dda76e8"
server_key_iv: "e45a41209f21a9200167c6adfecf74ea"
server_key_version: "encryptionkey@0"
I20260925 15:15:25.765348 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "be6f43b4cc884646a2eaf0ea5523580b"
format_stamp: "Formatted at 2026-09-25 15:15:25 on dist-test-slave-c5d6"
server_key: "0e38f2d3c7e05276c64083187dda76e8"
server_key_iv: "e45a41209f21a9200167c6adfecf74ea"
server_key_version: "encryptionkey@0"
I20260925 15:15:25.766710 11899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:25.767629 11914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:25.767800 11899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:25.767900 11899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "be6f43b4cc884646a2eaf0ea5523580b"
format_stamp: "Formatted at 2026-09-25 15:15:25 on dist-test-slave-c5d6"
server_key: "0e38f2d3c7e05276c64083187dda76e8"
server_key_iv: "e45a41209f21a9200167c6adfecf74ea"
server_key_version: "encryptionkey@0"
I20260925 15:15:25.767975 11899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:25.792721 11899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:25.795181 11899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:25.795368 11899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:25.799624 11899 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:45527
I20260925 15:15:25.799669 11966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:45527 every 8 connection(s)
I20260925 15:15:25.800184 11899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:25.800895 11967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:25.802376 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 11899
I20260925 15:15:25.802489 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:25.802681 8366 external_mini_cluster.cc:1428] Setting key 2412d8f9edca785cec6aa93257f05cc2
I20260925 15:15:25.803532 11967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b: Bootstrap starting.
I20260925 15:15:25.804160 11967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:25.804502 11967 log.cc:824] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:25.805233 11967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b: No bootstrap required, opened a new log
I20260925 15:15:25.807452 11967 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6f43b4cc884646a2eaf0ea5523580b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45527 } }
I20260925 15:15:25.807658 11967 raft_consensus.cc:374] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:25.807714 11967 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6f43b4cc884646a2eaf0ea5523580b, State: Initialized, Role: FOLLOWER
I20260925 15:15:25.807850 11967 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6f43b4cc884646a2eaf0ea5523580b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45527 } }
I20260925 15:15:25.807950 11967 raft_consensus.cc:388] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:25.807982 11967 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:25.808029 11967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:25.808758 11967 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6f43b4cc884646a2eaf0ea5523580b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45527 } }
I20260925 15:15:25.808873 11967 leader_election.cc:302] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be6f43b4cc884646a2eaf0ea5523580b; no voters:
I20260925 15:15:25.809033 11967 leader_election.cc:288] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:25.809092 11972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:25.809278 11972 raft_consensus.cc:686] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [term 1 LEADER]: Becoming Leader. State: Replica: be6f43b4cc884646a2eaf0ea5523580b, State: Running, Role: LEADER
I20260925 15:15:25.809299 11967 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:25.809403 11972 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6f43b4cc884646a2eaf0ea5523580b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45527 } }
Sep 25 15:15:25 dist-test-slave-c5d6 krb5kdc[11886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:25.809754 11972 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6f43b4cc884646a2eaf0ea5523580b. Latest consensus state: current_term: 1 leader_uuid: "be6f43b4cc884646a2eaf0ea5523580b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6f43b4cc884646a2eaf0ea5523580b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45527 } } }
I20260925 15:15:25.809821 11972 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:25.809960 11973 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be6f43b4cc884646a2eaf0ea5523580b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6f43b4cc884646a2eaf0ea5523580b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45527 } } }
I20260925 15:15:25.810102 11973 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:25.810456 11979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:25.811002 11979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:25.813246 11979 catalog_manager.cc:1347] Generated new cluster ID: 147b1cd7531d4c03ae281a343c67f0d4
I20260925 15:15:25.813320 11979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:25.814515 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:25.803209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.8.43.190:45527)
0925 15:15:25.803404 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:25.803412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:25.803436 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:15:25.803905 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:25.803911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:25.803959 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:25.804077 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:25.804855 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:25.805819 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:25.806873 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:25.807344 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:25.807435 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:25.810464 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:25.810468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:25.810468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:25.810480 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:25.812500 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:25.812929 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:25.812932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:25.812933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:25.813032 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:25.813256 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:25.813257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:25.813257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:25.813579 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:25.813728 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:25.814179 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 15:15:25.814300 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260925 15:15:25.822088 11979 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:25.822655 11979 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:25.836797 11979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be6f43b4cc884646a2eaf0ea5523580b: Generated new TSK 0
I20260925 15:15:25.836999 11979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:25.862135 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45527
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:25.945827 11993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:25.946018 11993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:25.946039 11993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:25.947182 11993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:25.947244 11993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:25.947273 11993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:25.948659 11993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:25.948827 11993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:25.948997 11993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:25.953702 11999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:25.953712 11998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:25.953748 11993 server_base.cc:1034] running on GCE node
W20260925 15:15:25.953729 12001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:25.954128 11993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:25.954372 11993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:25.955539 11993 hybrid_clock.cc:648] HybridClock initialized: now 1790349325955535 us; error 48 us; skew 500 ppm
Sep 25 15:15:25 dist-test-slave-c5d6 krb5kdc[11886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:25.962467 11993 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:25.962925 11993 webserver.cc:460] Webserver started at http://127.8.43.129:35185/ using document root <none> and password file <none>
I20260925 15:15:25.963171 11993 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:25.963265 11993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:25.963413 11993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:25.964591 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "370fc05ce29c4f3283d58b74171f75a0"
format_stamp: "Formatted at 2026-09-25 15:15:25 on dist-test-slave-c5d6"
server_key: "b3fc339b5919019f89d9a1a91626f51c"
server_key_iv: "5b045c38abc73fddf7b1a3b219ab3066"
server_key_version: "encryptionkey@0"
I20260925 15:15:25.964964 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "370fc05ce29c4f3283d58b74171f75a0"
format_stamp: "Formatted at 2026-09-25 15:15:25 on dist-test-slave-c5d6"
server_key: "b3fc339b5919019f89d9a1a91626f51c"
server_key_iv: "5b045c38abc73fddf7b1a3b219ab3066"
server_key_version: "encryptionkey@0"
I20260925 15:15:25.966224 11993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:25.966985 12008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:25.967193 11993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:25.967334 11993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "370fc05ce29c4f3283d58b74171f75a0"
format_stamp: "Formatted at 2026-09-25 15:15:25 on dist-test-slave-c5d6"
server_key: "b3fc339b5919019f89d9a1a91626f51c"
server_key_iv: "5b045c38abc73fddf7b1a3b219ab3066"
server_key_version: "encryptionkey@0"
I20260925 15:15:25.967406 11993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:25.990222 11993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:25.992311 11993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:25.992458 11993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:25.992707 11993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:25.993029 11993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:25.993081 11993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:25.993115 11993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:25.993144 11993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:25.998973 11993 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:40709
I20260925 15:15:25.999022 12121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:40709 every 8 connection(s)
I20260925 15:15:25.999418 11993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[11886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:26.007071 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 11993
I20260925 15:15:26.007194 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:26.007398 8366 external_mini_cluster.cc:1428] Setting key 99d619b173332bb5a3f38b833c0cdf36
I20260925 15:15:26.010491 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:26.000179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:37635 (local address 127.8.43.190:45527)
0925 15:15:26.000252 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:26.000254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:26.000437 (+ 183us) server_negotiation.cc:407] Connection header received
0925 15:15:26.001328 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:26.001329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:26.001338 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:26.001430 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:26.002343 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.003162 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.004332 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.004581 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.004661 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:26.006819 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:26.006820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:26.006821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:26.006829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:26.008672 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.009177 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.009178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.009178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.009242 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.009580 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.009581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.009581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.009718 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:26.009811 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:26.010231 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 15:15:26.010341 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:15:26.010586 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:26.000151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:45527 (local address 127.8.43.129:37635)
0925 15:15:26.000305 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:26.000348 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:26.001238 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:26.001569 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:26.001572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:26.001942 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:26.002220 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:26.002227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.003274 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.003275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:26.004219 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:26.004221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.004672 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.004672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:26.004771 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:26.005207 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:26.005214 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:26.006707 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:26.008760 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:26.008767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:26.008769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:26.009097 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:26.009341 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:26.009342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:26.009342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:26.009494 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:26.009802 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:26.009809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:26.009980 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 15:15:26.010201 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 15:15:26.010357 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260925 15:15:26.011207 12122 heartbeater.cc:347] Connected to a master server at 127.8.43.190:45527
I20260925 15:15:26.011346 12122 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:26.011557 12122 heartbeater.cc:511] Master 127.8.43.190:45527 requested a full tablet report, sending...
I20260925 15:15:26.012173 11931 ts_manager.cc:194] Registered new tserver with Master: 370fc05ce29c4f3283d58b74171f75a0 (127.8.43.129:40709)
I20260925 15:15:26.012862 11931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:37635
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:26.034078 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45527
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:26.115386 12127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:26.115674 12127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:26.115713 12127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:26.116848 12127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:26.116910 12127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:26.116962 12127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:26.118477 12127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:26.118728 12127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:26.118957 12127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:26.123878 12135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:26.123984 12133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:26.124042 12127 server_base.cc:1034] running on GCE node
W20260925 15:15:26.123869 12132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:26.124577 12127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:26.124843 12127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:26.125995 12127 hybrid_clock.cc:648] HybridClock initialized: now 1790349326125981 us; error 42 us; skew 500 ppm
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[11886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:26.128437 12127 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:26.128901 12127 webserver.cc:460] Webserver started at http://127.8.43.130:44529/ using document root <none> and password file <none>
I20260925 15:15:26.129142 12127 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:26.129209 12127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:26.129369 12127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:26.130429 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "1714b297ab32412fba19708a07beda7c"
format_stamp: "Formatted at 2026-09-25 15:15:26 on dist-test-slave-c5d6"
server_key: "bae0e20056cb72bd13cc3c1355c30af9"
server_key_iv: "8c7382c77dd309c0cdfde41f1b9689f6"
server_key_version: "encryptionkey@0"
I20260925 15:15:26.130788 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "1714b297ab32412fba19708a07beda7c"
format_stamp: "Formatted at 2026-09-25 15:15:26 on dist-test-slave-c5d6"
server_key: "bae0e20056cb72bd13cc3c1355c30af9"
server_key_iv: "8c7382c77dd309c0cdfde41f1b9689f6"
server_key_version: "encryptionkey@0"
I20260925 15:15:26.132148 12127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:26.132903 12142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:26.133083 12127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:26.133173 12127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "1714b297ab32412fba19708a07beda7c"
format_stamp: "Formatted at 2026-09-25 15:15:26 on dist-test-slave-c5d6"
server_key: "bae0e20056cb72bd13cc3c1355c30af9"
server_key_iv: "8c7382c77dd309c0cdfde41f1b9689f6"
server_key_version: "encryptionkey@0"
I20260925 15:15:26.133252 12127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:26.140578 12127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:26.142155 12127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:26.142305 12127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:26.142517 12127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:26.142787 12127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:26.142838 12127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:26.142870 12127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:26.142896 12127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:26.148613 12127 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:43227
I20260925 15:15:26.148658 12255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:43227 every 8 connection(s)
I20260925 15:15:26.148976 12127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:26.150197 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 12127
I20260925 15:15:26.150306 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:26.150470 8366 external_mini_cluster.cc:1428] Setting key 90cac82a7ce1589739e616397fe920d3
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[11886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:26.158980 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:26.149701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:37047 (local address 127.8.43.190:45527)
0925 15:15:26.149744 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:26.149746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:26.149918 (+ 172us) server_negotiation.cc:407] Connection header received
0925 15:15:26.150892 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:26.150901 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:26.150915 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:26.150965 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:26.151635 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.152358 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.153855 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.154163 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.154230 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:26.156117 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:26.156119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:26.156119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:26.156127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:26.157715 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.158086 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.158087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.158087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.158143 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.158365 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.158366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.158366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.158516 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:26.158597 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:26.158813 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 15:15:26.158891 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:15:26.159027 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:26.149648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:45527 (local address 127.8.43.130:37047)
0925 15:15:26.149799 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:26.149837 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:26.150777 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:26.151035 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:26.151039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:26.151361 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:26.151542 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:26.151546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.152477 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.152478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:26.153763 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:26.153765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.154236 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.154237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:26.154322 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:26.154760 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:26.154767 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:26.156032 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:26.157797 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:26.157802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:26.157804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:26.158012 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:26.158216 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:26.158217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:26.158217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:26.158300 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:26.158590 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:26.158593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:26.158691 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 15:15:26.158767 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:15:26.158871 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
I20260925 15:15:26.159749 12256 heartbeater.cc:347] Connected to a master server at 127.8.43.190:45527
I20260925 15:15:26.159821 12256 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:26.159943 12256 heartbeater.cc:511] Master 127.8.43.190:45527 requested a full tablet report, sending...
I20260925 15:15:26.160336 11931 ts_manager.cc:194] Registered new tserver with Master: 1714b297ab32412fba19708a07beda7c (127.8.43.130:43227)
I20260925 15:15:26.160809 11931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:37047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:26.173636 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45527
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:26.254750 12261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:26.254997 12261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:26.255035 12261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:26.256282 12261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:26.256351 12261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:26.256397 12261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:26.257933 12261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:26.258194 12261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:26.258435 12261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:26.263430 12269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:26.263422 12266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:26.263506 12261 server_base.cc:1034] running on GCE node
W20260925 15:15:26.263419 12267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:26.263870 12261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:26.264163 12261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:26.265339 12261 hybrid_clock.cc:648] HybridClock initialized: now 1790349326265326 us; error 24 us; skew 500 ppm
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[11886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:26.267853 12261 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:26.268290 12261 webserver.cc:460] Webserver started at http://127.8.43.131:45667/ using document root <none> and password file <none>
I20260925 15:15:26.268472 12261 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:26.268512 12261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:26.268600 12261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:26.269591 12261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "dd69520c2bc84a1fa2931555a5b60abf"
format_stamp: "Formatted at 2026-09-25 15:15:26 on dist-test-slave-c5d6"
server_key: "c916bd7d2bc6c90e1d92b2ed5fdd4071"
server_key_iv: "6f7190e3ccdc791d5aa2ed067ba11b8b"
server_key_version: "encryptionkey@0"
I20260925 15:15:26.269905 12261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "dd69520c2bc84a1fa2931555a5b60abf"
format_stamp: "Formatted at 2026-09-25 15:15:26 on dist-test-slave-c5d6"
server_key: "c916bd7d2bc6c90e1d92b2ed5fdd4071"
server_key_iv: "6f7190e3ccdc791d5aa2ed067ba11b8b"
server_key_version: "encryptionkey@0"
I20260925 15:15:26.271384 12261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:26.272181 12276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:26.272367 12261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:26.272449 12261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "dd69520c2bc84a1fa2931555a5b60abf"
format_stamp: "Formatted at 2026-09-25 15:15:26 on dist-test-slave-c5d6"
server_key: "c916bd7d2bc6c90e1d92b2ed5fdd4071"
server_key_iv: "6f7190e3ccdc791d5aa2ed067ba11b8b"
server_key_version: "encryptionkey@0"
I20260925 15:15:26.272511 12261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:26.297986 12261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:26.299777 12261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:26.299885 12261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:26.300068 12261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:26.300392 12261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:26.300426 12261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:26.300448 12261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:26.300509 12261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:26.306269 12261 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:32911
I20260925 15:15:26.306340 12389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:32911 every 8 connection(s)
I20260925 15:15:26.306703 12261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:26.307935 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 12261
I20260925 15:15:26.308035 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:26.308230 8366 external_mini_cluster.cc:1428] Setting key e33c975701ece32437b898c775f76a5b
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[11886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:26.316138 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:26.307473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:56031 (local address 127.8.43.190:45527)
0925 15:15:26.307515 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:26.307517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:26.307745 (+ 228us) server_negotiation.cc:407] Connection header received
0925 15:15:26.309003 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:26.309005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:26.309016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:26.309067 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:26.309682 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.310267 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.311330 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.311545 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.311617 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:26.313649 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:26.313650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:26.313650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:26.313656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:26.314850 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.315258 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.315259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.315260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.315319 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.315550 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.315551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.315551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.315648 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:26.315713 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:26.316018 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:15:26.316076 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:26.316308 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:26.307453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:45527 (local address 127.8.43.131:56031)
0925 15:15:26.307610 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:26.307650 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:26.308888 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:26.309138 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:26.309143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:26.309407 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:26.309589 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:26.309594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.310357 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.310358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:26.311198 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:26.311200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.311635 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.311636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:26.311754 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:26.312188 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:26.312196 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:26.313568 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:26.314923 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:26.314929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:26.314930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:26.315141 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:26.315375 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:26.315376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:26.315376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:26.315470 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:26.315727 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:26.315732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:26.315910 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 15:15:26.315992 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:15:26.316108 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":101,"threads_started":1}
I20260925 15:15:26.316773 12390 heartbeater.cc:347] Connected to a master server at 127.8.43.190:45527
I20260925 15:15:26.316877 12390 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:26.317066 12390 heartbeater.cc:511] Master 127.8.43.190:45527 requested a full tablet report, sending...
I20260925 15:15:26.317493 11931 ts_manager.cc:194] Registered new tserver with Master: dd69520c2bc84a1fa2931555a5b60abf (127.8.43.131:32911)
I20260925 15:15:26.317818 11931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:56031
I20260925 15:15:26.319885 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[11886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[11886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:26.342542 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:26.334434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.8.43.129:40709)
0925 15:15:26.334636 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:26.334640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:26.334650 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:15:26.334675 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:26.334676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:26.334690 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:26.334759 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:26.335483 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.336241 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.337302 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.337638 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.337699 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:26.339992 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:26.339999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:26.339999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:26.340006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:26.341494 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.341801 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.341804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.341804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.341869 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.342051 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.342052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.342052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.342189 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:26.342304 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:26.342377 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:15:26.342436 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[11886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:26.350696 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:26.344170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.8.43.190:45527)
0925 15:15:26.344224 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:26.344225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:26.344234 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:26.344278 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:26.344278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:26.344288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:26.344327 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:26.344795 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.345267 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.345953 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:26.346122 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:26.346169 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:26.348454 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:26.348456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:26.348457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:26.348463 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:26.349695 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.349992 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.349993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.349993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.350038 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:26.350189 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:26.350190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:26.350190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:26.350314 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:26.350392 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:26.350575 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:15:26.350634 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 15:15:26.350947 11931 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48360
I20260925 15:15:26.351307 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 11993
I20260925 15:15:26.356097 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 12127
I20260925 15:15:26.360391 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 12261
I20260925 15:15:26.364392 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 11899
2026-09-25T15:15:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1857 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[12402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[12402](info): set up 2 sockets
Sep 25 15:15:26 dist-test-slave-c5d6 krb5kdc[12402](info): commencing operation
krb5kdc: starting...
W20260925 15:15:27.403177 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:27 dist-test-slave-c5d6 krb5kdc[12402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:27.516263 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:36321
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:36321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:27.594493 12415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:27.594691 12415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:27.594717 12415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:27.595830 12415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:27.595865 12415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:27.595878 12415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:27.595891 12415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:27.595903 12415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:27.597319 12415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:36321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:27.597510 12415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:27.597672 12415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:27.602341 12421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:27.602335 12420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:27.602545 12415 server_base.cc:1034] running on GCE node
W20260925 15:15:27.602330 12423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:27.602914 12415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:27.603282 12415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:27.604494 12415 hybrid_clock.cc:648] HybridClock initialized: now 1790349327604481 us; error 25 us; skew 500 ppm
Sep 25 15:15:27 dist-test-slave-c5d6 krb5kdc[12402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349327, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:27.606930 12415 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:27.607336 12415 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:27.607414 12415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:27.607546 12415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:27.608604 12415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "1006c992f2bd4c858d733264080c2e89"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "36d2500348f77839c0b3376a1dbc9ef4"
server_key_iv: "a92e27dd55cff8899b48859061484dbc"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.608969 12415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "1006c992f2bd4c858d733264080c2e89"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "36d2500348f77839c0b3376a1dbc9ef4"
server_key_iv: "a92e27dd55cff8899b48859061484dbc"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.610213 12415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:15:27.610924 12429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:27.611107 12415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:27.611204 12415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "1006c992f2bd4c858d733264080c2e89"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "36d2500348f77839c0b3376a1dbc9ef4"
server_key_iv: "a92e27dd55cff8899b48859061484dbc"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.611311 12415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:27.619589 12415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:27.621809 12415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:27.621975 12415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:27.625808 12415 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:36321
I20260925 15:15:27.625830 12481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:36321 every 8 connection(s)
I20260925 15:15:27.626113 12415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:27.627154 12482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:27.629434 12482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89: Bootstrap starting.
I20260925 15:15:27.629951 12482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:27.630203 12482 log.cc:824] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:27.630625 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 12415
I20260925 15:15:27.630735 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:27.630904 8366 external_mini_cluster.cc:1428] Setting key 1cf87a2962dd5213ea991d403796b4de
I20260925 15:15:27.630930 12482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89: No bootstrap required, opened a new log
I20260925 15:15:27.632776 12482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1006c992f2bd4c858d733264080c2e89" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36321 } }
I20260925 15:15:27.632934 12482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:27.632970 12482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1006c992f2bd4c858d733264080c2e89, State: Initialized, Role: FOLLOWER
I20260925 15:15:27.633101 12482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1006c992f2bd4c858d733264080c2e89" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36321 } }
I20260925 15:15:27.633184 12482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:27.633219 12482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:27.633267 12482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:27.634268 12482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1006c992f2bd4c858d733264080c2e89" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36321 } }
I20260925 15:15:27.634352 12482 leader_election.cc:302] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1006c992f2bd4c858d733264080c2e89; no voters:
I20260925 15:15:27.634543 12482 leader_election.cc:288] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:27.634577 12487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:27.634672 12487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [term 1 LEADER]: Becoming Leader. State: Replica: 1006c992f2bd4c858d733264080c2e89, State: Running, Role: LEADER
I20260925 15:15:27.634756 12487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1006c992f2bd4c858d733264080c2e89" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36321 } }
I20260925 15:15:27.634786 12482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:27.635077 12487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1006c992f2bd4c858d733264080c2e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1006c992f2bd4c858d733264080c2e89" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36321 } } }
I20260925 15:15:27.635133 12487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:27.635293 12487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1006c992f2bd4c858d733264080c2e89. Latest consensus state: current_term: 1 leader_uuid: "1006c992f2bd4c858d733264080c2e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1006c992f2bd4c858d733264080c2e89" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36321 } } }
I20260925 15:15:27.635355 12487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:27.635866 12492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:27.636449 12492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:27.638062 12492 catalog_manager.cc:1347] Generated new cluster ID: 7ed20c2cf80c42ff98d41261b9a708d9
I20260925 15:15:27.638118 12492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:15:27 dist-test-slave-c5d6 krb5kdc[12402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:27.643208 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:27.631452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.8.43.190:36321)
0925 15:15:27.631606 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:27.631615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:27.631648 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:15:27.632762 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:27.632768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:27.632815 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:27.632923 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:27.633700 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.634695 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.635861 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.636505 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.636613 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:27.639678 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:27.639681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:27.639682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:27.639693 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:27.641566 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:27.641963 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:27.641965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:27.641966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:27.642079 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:27.642288 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:27.642289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:27.642289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:27.642586 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:27.642732 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:27.642893 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 15:15:27.643020 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260925 15:15:27.650516 12492 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:27.651275 12492 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:27.659438 12492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89: Generated new TSK 0
I20260925 15:15:27.659654 12492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:27.686921 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36321
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:27.768725 12508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:27.768972 12508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:27.769016 12508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:27.770210 12508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:27.770295 12508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:27.770341 12508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:27.771960 12508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:27.772235 12508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:27.772480 12508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:27.777382 12516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:27.777377 12514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:27.777518 12508 server_base.cc:1034] running on GCE node
W20260925 15:15:27.777679 12513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:27.777963 12508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:27.778249 12508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:27.779405 12508 hybrid_clock.cc:648] HybridClock initialized: now 1790349327779384 us; error 41 us; skew 500 ppm
Sep 25 15:15:27 dist-test-slave-c5d6 krb5kdc[12402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349327, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:27.782169 12508 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:27.782562 12508 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:27.782634 12508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:27.782788 12508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:27.783975 12508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "9e6116722006448f848fff80db8c0969"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "17785450cb24032df700373aa2326a3a"
server_key_iv: "994e2d5e7db04760df5932699d323888"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.784353 12508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "9e6116722006448f848fff80db8c0969"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "17785450cb24032df700373aa2326a3a"
server_key_iv: "994e2d5e7db04760df5932699d323888"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.785673 12508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:27.786381 12522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:27.786548 12508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:27.786646 12508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "9e6116722006448f848fff80db8c0969"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "17785450cb24032df700373aa2326a3a"
server_key_iv: "994e2d5e7db04760df5932699d323888"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.786739 12508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:27.807336 12508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:27.809216 12508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:27.809376 12508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:27.809605 12508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:27.809911 12508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:27.809962 12508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:27.810041 12508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:27.810073 12508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:27.816200 12508 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:46777
I20260925 15:15:27.816251 12635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:46777 every 8 connection(s)
I20260925 15:15:27.816557 12508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:27.822184 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 12508
I20260925 15:15:27.822319 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:27.822541 8366 external_mini_cluster.cc:1428] Setting key 3d527e7ae10e2907dd2a1d1088184010
Sep 25 15:15:27 dist-test-slave-c5d6 krb5kdc[12402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349327, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:27.826339 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:27.817250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:43327 (local address 127.8.43.190:36321)
0925 15:15:27.817294 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:27.817296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:27.817560 (+ 264us) server_negotiation.cc:407] Connection header received
0925 15:15:27.818086 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:27.818088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:27.818105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:27.818164 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:27.819254 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.819803 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.820824 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.821061 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.821151 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:27.823386 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:27.823387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:27.823388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:27.823394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:27.824952 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:27.825353 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:27.825354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:27.825354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:27.825421 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:27.825652 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:27.825653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:27.825654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:27.825803 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:27.825889 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:27.826164 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:15:27.826249 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:27.826555 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:27.817244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36321 (local address 127.8.43.129:43327)
0925 15:15:27.817428 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:27.817465 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:27.817978 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:27.818241 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:27.818246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:27.818525 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:27.819116 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:27.819121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.819912 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.819913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:27.820732 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:27.820735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.821160 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.821161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:27.821283 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:27.821724 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:27.821733 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:27.823298 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:27.825036 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:27.825042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:27.825043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:27.825276 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:27.825488 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:27.825489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:27.825489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:27.825587 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:27.825881 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:27.825888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:27.826033 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:15:27.826142 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:15:27.826298 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260925 15:15:27.827036 12636 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36321
I20260925 15:15:27.827144 12636 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:27.827325 12636 heartbeater.cc:511] Master 127.8.43.190:36321 requested a full tablet report, sending...
I20260925 15:15:27.827906 12446 ts_manager.cc:194] Registered new tserver with Master: 9e6116722006448f848fff80db8c0969 (127.8.43.129:46777)
I20260925 15:15:27.828897 12446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:43327
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:27.847481 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36321
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:27.934952 12641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:27.935178 12641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:27.935206 12641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:27.936568 12641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:27.936616 12641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:27.936646 12641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:27.938687 12641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:27.938897 12641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:27.939092 12641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:27.944046 12647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:27.944108 12649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:27.944051 12646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:27.944236 12641 server_base.cc:1034] running on GCE node
I20260925 15:15:27.944542 12641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:27.944742 12641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:27.945892 12641 hybrid_clock.cc:648] HybridClock initialized: now 1790349327945875 us; error 36 us; skew 500 ppm
Sep 25 15:15:27 dist-test-slave-c5d6 krb5kdc[12402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349327, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:27.948450 12641 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:27.948802 12641 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:27.948851 12641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:27.948987 12641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:27.950021 12641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "21fe7bc302cb4b36945030cb29e2caed"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "b510241dcd70cbffabbbf0cf4a482734"
server_key_iv: "efc95d8f8fd273cae395a4ba73411e13"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.950376 12641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "21fe7bc302cb4b36945030cb29e2caed"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "b510241dcd70cbffabbbf0cf4a482734"
server_key_iv: "efc95d8f8fd273cae395a4ba73411e13"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.951788 12641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:27.952584 12655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:27.952764 12641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:15:27.952863 12641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "21fe7bc302cb4b36945030cb29e2caed"
format_stamp: "Formatted at 2026-09-25 15:15:27 on dist-test-slave-c5d6"
server_key: "b510241dcd70cbffabbbf0cf4a482734"
server_key_iv: "efc95d8f8fd273cae395a4ba73411e13"
server_key_version: "encryptionkey@0"
I20260925 15:15:27.952940 12641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:27.967125 12641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:27.969080 12641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:27.969247 12641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:27.969480 12641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:27.969794 12641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:27.969846 12641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:27.969882 12641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:27.969907 12641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:27.975796 12641 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:38599
I20260925 15:15:27.975836 12768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:38599 every 8 connection(s)
I20260925 15:15:27.976101 12641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:27.981781 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 12641
I20260925 15:15:27.981906 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
Sep 25 15:15:27 dist-test-slave-c5d6 krb5kdc[12402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349327, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:27.982102 8366 external_mini_cluster.cc:1428] Setting key 9f3a0e37e75ae1d58191dae560620d1e
I20260925 15:15:27.985548 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:27.976793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:34183 (local address 127.8.43.190:36321)
0925 15:15:27.976835 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:27.976837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:27.977094 (+ 257us) server_negotiation.cc:407] Connection header received
0925 15:15:27.977625 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:27.977627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:27.977639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:27.977690 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:27.978638 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.979141 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.980153 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.980341 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.980425 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:27.982670 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:27.982671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:27.982672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:27.982679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:27.984135 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:27.984562 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:27.984563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:27.984564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:27.984621 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:27.984885 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:27.984886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:27.984887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:27.985008 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:27.985096 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:27.985373 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:15:27.985457 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:15:27.985874 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:27.976803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36321 (local address 127.8.43.130:34183)
0925 15:15:27.976970 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:27.977005 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:27.977520 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:27.977771 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:27.977775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:27.978045 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:27.978536 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:27.978541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.979251 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.979252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:27.980052 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:27.980054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:27.980430 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:27.980431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:27.980560 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:27.981007 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:27.981014 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:27.982564 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:27.984241 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:27.984248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:27.984250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:27.984471 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:27.984692 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:27.984693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:27.984694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:27.984818 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:27.985089 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:27.985094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:27.985235 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:15:27.985354 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:15:27.985511 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260925 15:15:27.986423 12769 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36321
I20260925 15:15:27.986541 12769 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:27.986706 12769 heartbeater.cc:511] Master 127.8.43.190:36321 requested a full tablet report, sending...
I20260925 15:15:27.987108 12446 ts_manager.cc:194] Registered new tserver with Master: 21fe7bc302cb4b36945030cb29e2caed (127.8.43.130:38599)
I20260925 15:15:27.987694 12446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:34183
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:28.008008 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36321
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:28.088485 12774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:28.088670 12774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:28.088691 12774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:28.089856 12774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:28.089895 12774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:28.089952 12774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:28.091387 12774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:28.091558 12774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:28.091714 12774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:28.096338 12779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:28.096413 12782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:28.096436 12780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:28.096799 12774 server_base.cc:1034] running on GCE node
I20260925 15:15:28.097054 12774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:28.097294 12774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:28.098436 12774 hybrid_clock.cc:648] HybridClock initialized: now 1790349328098428 us; error 41 us; skew 500 ppm
Sep 25 15:15:28 dist-test-slave-c5d6 krb5kdc[12402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349328, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:28.100917 12774 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:28.101271 12774 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:28.101337 12774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:28.101469 12774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:28.102451 12774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "2b43f4ac81474bacb1f36542713f3809"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "31c88fee31bbc51ddbf89c80dde9cbfb"
server_key_iv: "b1aef8d400ffaa9f578ec781f96edc6b"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.102849 12774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "2b43f4ac81474bacb1f36542713f3809"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "31c88fee31bbc51ddbf89c80dde9cbfb"
server_key_iv: "b1aef8d400ffaa9f578ec781f96edc6b"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.104166 12774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:28.104900 12788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.105073 12774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:28.105166 12774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "2b43f4ac81474bacb1f36542713f3809"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "31c88fee31bbc51ddbf89c80dde9cbfb"
server_key_iv: "b1aef8d400ffaa9f578ec781f96edc6b"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.105244 12774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:28.143054 12774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:28.145200 12774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:28.145367 12774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:28.145635 12774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:28.145947 12774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:28.146026 12774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.146068 12774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:28.146090 12774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.151705 12774 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:40779
I20260925 15:15:28.151746 12901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:40779 every 8 connection(s)
I20260925 15:15:28.152015 12774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:28.152861 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 12774
I20260925 15:15:28.152961 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:28.153167 8366 external_mini_cluster.cc:1428] Setting key 1be2a5c41b91ef37f1d2b6aaf7c3e1d1
Sep 25 15:15:28 dist-test-slave-c5d6 krb5kdc[12402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349328, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:28.161425 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.152661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:45655 (local address 127.8.43.190:36321)
0925 15:15:28.152723 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.152725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.152945 (+ 220us) server_negotiation.cc:407] Connection header received
0925 15:15:28.153647 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.153648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.153661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.153720 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.154734 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.155291 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.156361 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.156598 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.156650 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.158620 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.158621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.158621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:28.158628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.160168 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:28.160527 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:28.160528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:28.160528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:28.160578 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:28.160814 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:28.160815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:28.160816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:28.160930 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.161042 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.161288 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.161360 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:15:28.161578 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.152664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:36321 (local address 127.8.43.131:45655)
0925 15:15:28.152812 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.152853 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.153499 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.153804 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.153808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.154172 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.154656 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.154662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.155399 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.155400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.156272 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.156274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.156672 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.156673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.156792 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.157235 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:28.157243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.158539 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.160235 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:28.160240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:28.160242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:28.160445 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:28.160637 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:28.160638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:28.160638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:28.160729 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:28.161043 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.161047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.161163 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.161248 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.161364 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 15:15:28.161978 12902 heartbeater.cc:347] Connected to a master server at 127.8.43.190:36321
I20260925 15:15:28.162089 12902 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:28.162261 12902 heartbeater.cc:511] Master 127.8.43.190:36321 requested a full tablet report, sending...
I20260925 15:15:28.162633 12446 ts_manager.cc:194] Registered new tserver with Master: 2b43f4ac81474bacb1f36542713f3809 (127.8.43.131:40779)
I20260925 15:15:28.163019 12446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:45655
I20260925 15:15:28.164852 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:15:28.172194 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.166116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.8.43.190:36321)
0925 15:15:28.166199 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.166201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.166267 (+ 66us) server_negotiation.cc:407] Connection header received
0925 15:15:28.166342 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.166343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.166352 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.166406 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.166970 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.167513 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.168241 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.168439 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.168508 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.169598 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.169599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.169600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:28.169607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.170803 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:28.171266 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:28.171267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:28.171268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:28.171335 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:28.171574 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:28.171575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:28.171576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:28.171757 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.171872 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.172004 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.172087 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:15:28.173763 12446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:28.175468 12446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:28.182053 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.177870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46777 (local address 127.0.0.1:46176)
0925 15:15:28.178070 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.178125 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.178194 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.178543 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.178546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.178548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:28.178612 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.178613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.179677 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.179679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.180855 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.180857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.181733 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.181734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.181835 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.181851 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.181914 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.181962 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260925 15:15:28.182224 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.177894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:38599 (local address 127.0.0.1:51560)
0925 15:15:28.178211 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.178222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.178273 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.178670 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.178671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.178674 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:28.178720 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.178722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.180072 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.180073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.181034 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.181036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.181950 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.181951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.182034 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.182038 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.182095 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.182139 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":106,"threads_started":1}
I20260925 15:15:28.182286 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.178199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.8.43.129:46777)
0925 15:15:28.178376 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.178393 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.178403 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:15:28.178445 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.178447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.178463 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.178541 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:28.178668 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.179583 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.180948 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.181641 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.181725 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.181782 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.182173 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.182215 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260925 15:15:28.182343 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.178014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.8.43.130:38599)
0925 15:15:28.178427 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.178431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.178441 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:15:28.178482 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.178484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.178501 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.178803 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:28.178883 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.179982 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.181107 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.181835 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.181946 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.182019 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.182107 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.182254 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":72,"threads_started":1}
I20260925 15:15:28.183019 12570 tablet_service.cc:1467] Processing CreateTablet for tablet 09f72210209942b2b2019a5b6ca6256d (DEFAULT_TABLE table=test-table [id=d8fcf912df814cd2aeb0c96fadcba96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:28.183019 12703 tablet_service.cc:1467] Processing CreateTablet for tablet 09f72210209942b2b2019a5b6ca6256d (DEFAULT_TABLE table=test-table [id=d8fcf912df814cd2aeb0c96fadcba96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:28.183367 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.178106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:40779 (local address 127.0.0.1:35844)
0925 15:15:28.178368 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.178378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.178433 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.178685 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.178686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.178688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:28.178721 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.178722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.181099 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.181099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.181750 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.181751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.183103 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.183104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.183166 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.183170 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.183265 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.183307 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":59,"threads_started":1}
I20260925 15:15:28.183420 12570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f72210209942b2b2019a5b6ca6256d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:28.183420 12703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f72210209942b2b2019a5b6ca6256d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:28.183619 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.178211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.8.43.131:40779)
0925 15:15:28.178366 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.178371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.178426 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:15:28.178503 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.178505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.178521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.178604 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:28.178806 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.179964 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.181838 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.183013 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.183103 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.183152 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.183492 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.183538 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260925 15:15:28.186879 12920 tablet_bootstrap.cc:492] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969: Bootstrap starting.
I20260925 15:15:28.187119 12919 tablet_bootstrap.cc:492] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: Bootstrap starting.
I20260925 15:15:28.187552 12836 tablet_service.cc:1467] Processing CreateTablet for tablet 09f72210209942b2b2019a5b6ca6256d (DEFAULT_TABLE table=test-table [id=d8fcf912df814cd2aeb0c96fadcba96f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:28.187736 12919 tablet_bootstrap.cc:654] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:28.187772 12920 tablet_bootstrap.cc:654] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:28.187901 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f72210209942b2b2019a5b6ca6256d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:28.188416 12920 log.cc:824] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:28.188416 12919 log.cc:824] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:28.189194 12920 tablet_bootstrap.cc:492] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969: No bootstrap required, opened a new log
I20260925 15:15:28.189194 12919 tablet_bootstrap.cc:492] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: No bootstrap required, opened a new log
I20260925 15:15:28.189263 12920 ts_tablet_manager.cc:1397] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:28.189263 12919 ts_tablet_manager.cc:1397] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:28.190956 12923 tablet_bootstrap.cc:492] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: Bootstrap starting.
I20260925 15:15:28.191291 12920 raft_consensus.cc:348] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.191291 12919 raft_consensus.cc:348] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.191457 12920 raft_consensus.cc:374] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:28.191465 12919 raft_consensus.cc:374] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:28.191497 12920 raft_consensus.cc:729] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6116722006448f848fff80db8c0969, State: Initialized, Role: FOLLOWER
I20260925 15:15:28.191497 12919 raft_consensus.cc:729] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fe7bc302cb4b36945030cb29e2caed, State: Initialized, Role: FOLLOWER
I20260925 15:15:28.191633 12920 consensus_queue.cc:260] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.191633 12919 consensus_queue.cc:260] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.191727 12923 tablet_bootstrap.cc:654] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:28.191910 12920 ts_tablet_manager.cc:1428] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:28.191910 12919 ts_tablet_manager.cc:1428] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:28.192039 12769 heartbeater.cc:503] Master 127.8.43.190:36321 was elected leader, sending a full tablet report...
I20260925 15:15:28.192039 12636 heartbeater.cc:503] Master 127.8.43.190:36321 was elected leader, sending a full tablet report...
I20260925 15:15:28.192327 12923 log.cc:824] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:28.193310 12923 tablet_bootstrap.cc:492] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: No bootstrap required, opened a new log
I20260925 15:15:28.193398 12923 ts_tablet_manager.cc:1397] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:28.194891 12923 raft_consensus.cc:348] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.195024 12923 raft_consensus.cc:374] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:28.195048 12923 raft_consensus.cc:729] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b43f4ac81474bacb1f36542713f3809, State: Initialized, Role: FOLLOWER
I20260925 15:15:28.195164 12923 consensus_queue.cc:260] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.195497 12923 ts_tablet_manager.cc:1428] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:15:28.195497 12902 heartbeater.cc:503] Master 127.8.43.190:36321 was elected leader, sending a full tablet report...
W20260925 15:15:28.226763 12770 tablet.cc:2369] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:28.244949 12924 raft_consensus.cc:482] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:28.245083 12924 raft_consensus.cc:504] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.245385 12924 leader_election.cc:288] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e6116722006448f848fff80db8c0969 (127.8.43.129:46777), 2b43f4ac81474bacb1f36542713f3809 (127.8.43.131:40779)
I20260925 15:15:28.248821 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.245567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:40779 (local address 127.8.43.130:46213)
0925 15:15:28.245653 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.245666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.245734 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.245979 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.245980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.245981 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:28.246037 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.246039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.247030 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.247031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.247923 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.247926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.248549 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.248550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.248619 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.248622 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.248686 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.248730 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:15:28.248827 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.245673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:46213 (local address 127.8.43.131:40779)
0925 15:15:28.245731 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.245733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.245745 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:15:28.245824 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.245826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.245839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.245913 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:28.246133 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.246945 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.248023 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.248475 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.248533 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.248539 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.248696 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.248755 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:15:28.249060 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.245700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:55603 (local address 127.8.43.129:46777)
0925 15:15:28.245740 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.245742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.246066 (+ 324us) server_negotiation.cc:407] Connection header received
0925 15:15:28.246177 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.246178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.246191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.246239 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:28.246512 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.247231 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.248242 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.248673 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.248737 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.248744 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.248913 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.248998 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:15:28.249189 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f72210209942b2b2019a5b6ca6256d" candidate_uuid: "21fe7bc302cb4b36945030cb29e2caed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b43f4ac81474bacb1f36542713f3809" is_pre_election: true
I20260925 15:15:28.249362 12856 raft_consensus.cc:2454] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21fe7bc302cb4b36945030cb29e2caed in term 0.
I20260925 15:15:28.249378 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.245772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46777 (local address 127.8.43.130:55603)
0925 15:15:28.246005 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.246015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.246092 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.246247 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.246247 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.246249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:28.246326 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.246328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.247329 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.247330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.248154 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:28.248156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.248754 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.248755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:28.248826 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.248829 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.249236 (+ 407us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.249269 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":147,"threads_started":1}
I20260925 15:15:28.249605 12656 leader_election.cc:302] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21fe7bc302cb4b36945030cb29e2caed, 2b43f4ac81474bacb1f36542713f3809; no voters:
I20260925 15:15:28.249750 12924 raft_consensus.cc:2781] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:28.249812 12924 raft_consensus.cc:482] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:28.249846 12924 raft_consensus.cc:3037] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:28.249781 12590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f72210209942b2b2019a5b6ca6256d" candidate_uuid: "21fe7bc302cb4b36945030cb29e2caed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6116722006448f848fff80db8c0969" is_pre_election: true
I20260925 15:15:28.249938 12590 raft_consensus.cc:2454] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21fe7bc302cb4b36945030cb29e2caed in term 0.
I20260925 15:15:28.250562 12924 raft_consensus.cc:504] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.250710 12924 leader_election.cc:288] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [CANDIDATE]: Term 1 election: Requested vote from peers 9e6116722006448f848fff80db8c0969 (127.8.43.129:46777), 2b43f4ac81474bacb1f36542713f3809 (127.8.43.131:40779)
I20260925 15:15:28.250850 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f72210209942b2b2019a5b6ca6256d" candidate_uuid: "21fe7bc302cb4b36945030cb29e2caed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b43f4ac81474bacb1f36542713f3809"
I20260925 15:15:28.250946 12856 raft_consensus.cc:3037] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:28.251053 12590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f72210209942b2b2019a5b6ca6256d" candidate_uuid: "21fe7bc302cb4b36945030cb29e2caed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e6116722006448f848fff80db8c0969"
I20260925 15:15:28.251157 12590 raft_consensus.cc:3037] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:28.251762 12856 raft_consensus.cc:2454] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21fe7bc302cb4b36945030cb29e2caed in term 1.
I20260925 15:15:28.251926 12656 leader_election.cc:302] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21fe7bc302cb4b36945030cb29e2caed, 2b43f4ac81474bacb1f36542713f3809; no voters:
I20260925 15:15:28.252033 12924 raft_consensus.cc:2781] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:28.252077 12590 raft_consensus.cc:2454] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21fe7bc302cb4b36945030cb29e2caed in term 1.
I20260925 15:15:28.252292 12924 raft_consensus.cc:686] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 1 LEADER]: Becoming Leader. State: Replica: 21fe7bc302cb4b36945030cb29e2caed, State: Running, Role: LEADER
I20260925 15:15:28.252417 12924 consensus_queue.cc:237] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } }
I20260925 15:15:28.253252 12446 catalog_manager.cc:5495] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed reported cstate change: term changed from 0 to 1, leader changed from <none> to 21fe7bc302cb4b36945030cb29e2caed (127.8.43.130). New cstate: current_term: 1 leader_uuid: "21fe7bc302cb4b36945030cb29e2caed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21fe7bc302cb4b36945030cb29e2caed" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 38599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 } health_report { overall_health: UNKNOWN } } }
I20260925 15:15:28.259025 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.256715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.8.43.130:38599)
0925 15:15:28.256760 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.256761 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.256770 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:28.256844 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.256845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.256859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.256918 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:28.257029 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.257460 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.258372 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:28.258552 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:28.258599 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:28.258742 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:28.258828 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:28.258881 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.258926 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.258941 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:15:28.260794 12856 raft_consensus.cc:1261] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 1 FOLLOWER]: Refusing update from remote peer 21fe7bc302cb4b36945030cb29e2caed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:28.260795 12590 raft_consensus.cc:1261] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 1 FOLLOWER]: Refusing update from remote peer 21fe7bc302cb4b36945030cb29e2caed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:28.261118 12929 consensus_queue.cc:1035] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e6116722006448f848fff80db8c0969" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:28.261220 12924 consensus_queue.cc:1035] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b43f4ac81474bacb1f36542713f3809" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:28.264036 12934 mvcc.cc:205] Tried to move back new op lower bound from 7333270848553381888 to 7333270848522502144. Current Snapshot: MvccSnapshot[applied={T|T < 7333270848553381888}]
I20260925 15:15:28.264143 12933 mvcc.cc:205] Tried to move back new op lower bound from 7333270848553381888 to 7333270848522502144. Current Snapshot: MvccSnapshot[applied={T|T < 7333270848553381888}]
I20260925 15:15:28.265764 12445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:28.265875 12445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:28.266597 12445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1006c992f2bd4c858d733264080c2e89: Sending DeleteTablet for 3 replicas of tablet 09f72210209942b2b2019a5b6ca6256d
I20260925 15:15:28.266947 12703 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f72210209942b2b2019a5b6ca6256d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:28 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:51560
I20260925 15:15:28.267009 12570 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f72210209942b2b2019a5b6ca6256d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:28 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:46176
I20260925 15:15:28.267187 12942 tablet_replica.cc:330] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969: stopping tablet replica
I20260925 15:15:28.267200 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f72210209942b2b2019a5b6ca6256d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:28 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:35844
I20260925 15:15:28.267308 12942 raft_consensus.cc:2229] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:28.267412 12942 raft_consensus.cc:2258] T 09f72210209942b2b2019a5b6ca6256d P 9e6116722006448f848fff80db8c0969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:28.267436 12943 tablet_replica.cc:330] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: stopping tablet replica
I20260925 15:15:28.267586 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 12508
I20260925 15:15:28.267618 12943 raft_consensus.cc:2229] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:28.267777 12943 raft_consensus.cc:2258] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:28.271445 12941 tablet_replica.cc:330] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: stopping tablet replica
I20260925 15:15:28.271559 12941 raft_consensus.cc:2229] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:28.271678 12941 raft_consensus.cc:2258] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:28.272063 12941 ts_tablet_manager.cc:1905] T 09f72210209942b2b2019a5b6ca6256d P 21fe7bc302cb4b36945030cb29e2caed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:28.272063 12943 ts_tablet_manager.cc:1905] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:15:28.273257 12433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:46777 (error 108)
W20260925 15:15:28.273820 12913 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:28.273678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46777 (local address 127.0.0.1:46182)
0925 15:15:28.273725 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.273776 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:46777: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 15:15:28.273954 12943 ts_tablet_manager.cc:1918] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:28.274044 12943 log.cc:1196] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/09f72210209942b2b2019a5b6ca6256d
W20260925 15:15:28.274036 12433 catalog_manager.cc:4584] TS 9e6116722006448f848fff80db8c0969 (127.8.43.129:46777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09f72210209942b2b2019a5b6ca6256d: Network error: Client connection negotiation failed: client connection to 127.8.43.129:46777: connect: Connection reset by peer (error 104)
I20260925 15:15:28.274056 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 12641
I20260925 15:15:28.274312 12943 ts_tablet_manager.cc:1939] T 09f72210209942b2b2019a5b6ca6256d P 2b43f4ac81474bacb1f36542713f3809: Deleting consensus metadata
I20260925 15:15:28.276168 12430 catalog_manager.cc:4841] TS 2b43f4ac81474bacb1f36542713f3809 (127.8.43.131:40779): tablet 09f72210209942b2b2019a5b6ca6256d (table test-table [id=d8fcf912df814cd2aeb0c96fadcba96f]) successfully deleted
I20260925 15:15:28.279297 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.279138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:38599 (local address 127.0.0.1:51592)
0925 15:15:28.279193 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.279265 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.130:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:15:28.279332 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 12774
I20260925 15:15:28.283751 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 12415
2026-09-25T15:15:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1919 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T15:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:28Z Disabled control of system clock
I20260925 15:15:28.310837 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:34469
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:34469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:15:28.390873 12948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:28.391093 12948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:28.391116 12948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:28.392302 12948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:28.392350 12948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:28.392365 12948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:28.392379 12948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:28.392421 12948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:28.393774 12948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:34469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:28.394044 12948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:28.394369 12948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:28.399498 12953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:28.399521 12954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:28.399533 12948 server_base.cc:1034] running on GCE node
W20260925 15:15:28.399478 12956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:28.399902 12948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:28.400213 12948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:28.401396 12948 hybrid_clock.cc:648] HybridClock initialized: now 1790349328401379 us; error 30 us; skew 500 ppm
I20260925 15:15:28.402647 12948 webserver.cc:460] Webserver started at http://127.8.43.190:42753/ using document root <none> and password file <none>
I20260925 15:15:28.402927 12948 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:28.403005 12948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:28.403267 12948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:28.404410 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "90da8c3e30b64915b4a604605b77b7e2"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "43c93479d399c2d2c21c68ce1c7f4f4b"
server_key_iv: "1659a813fd6adf69f5c57cd1c13b183f"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.404824 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "90da8c3e30b64915b4a604605b77b7e2"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "43c93479d399c2d2c21c68ce1c7f4f4b"
server_key_iv: "1659a813fd6adf69f5c57cd1c13b183f"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.406189 12948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:28.406984 12962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.407184 12948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:28.407312 12948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "90da8c3e30b64915b4a604605b77b7e2"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "43c93479d399c2d2c21c68ce1c7f4f4b"
server_key_iv: "1659a813fd6adf69f5c57cd1c13b183f"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.407420 12948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:28.410787 12948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:28.411074 12948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:28.411226 12948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:28.415385 12948 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:34469
I20260925 15:15:28.415459 13014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:34469 every 8 connection(s)
I20260925 15:15:28.415762 12948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:28.416383 13015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:28.419152 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2: Bootstrap starting.
I20260925 15:15:28.419781 13015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:28.420077 13015 log.cc:824] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:28.420831 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2: No bootstrap required, opened a new log
I20260925 15:15:28.422379 13015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90da8c3e30b64915b4a604605b77b7e2" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34469 } }
I20260925 15:15:28.422518 13015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:28.422541 13015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90da8c3e30b64915b4a604605b77b7e2, State: Initialized, Role: FOLLOWER
I20260925 15:15:28.422689 13015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90da8c3e30b64915b4a604605b77b7e2" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34469 } }
I20260925 15:15:28.422752 13015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:28.422791 13015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:28.422840 13015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:28.423539 13015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90da8c3e30b64915b4a604605b77b7e2" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34469 } }
I20260925 15:15:28.423627 13015 leader_election.cc:302] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90da8c3e30b64915b4a604605b77b7e2; no voters:
I20260925 15:15:28.423821 13015 leader_election.cc:288] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:28.423908 13018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:28.424058 13018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [term 1 LEADER]: Becoming Leader. State: Replica: 90da8c3e30b64915b4a604605b77b7e2, State: Running, Role: LEADER
I20260925 15:15:28.424058 13015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:28.424168 13018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90da8c3e30b64915b4a604605b77b7e2" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34469 } }
I20260925 15:15:28.424546 13019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90da8c3e30b64915b4a604605b77b7e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90da8c3e30b64915b4a604605b77b7e2" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34469 } } }
I20260925 15:15:28.424633 13019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:28.424795 13020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90da8c3e30b64915b4a604605b77b7e2. Latest consensus state: current_term: 1 leader_uuid: "90da8c3e30b64915b4a604605b77b7e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90da8c3e30b64915b4a604605b77b7e2" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34469 } } }
I20260925 15:15:28.424880 13020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:28.424883 13025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:28.425264 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 12948
I20260925 15:15:28.425359 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:28.425576 8366 external_mini_cluster.cc:1428] Setting key 69e31e53f9b3e8f8e83642e436556561
I20260925 15:15:28.426065 13025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:28.427631 13025 catalog_manager.cc:1347] Generated new cluster ID: f0b8a6e3f4b94080be5d53681d5a83be
I20260925 15:15:28.427682 13025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:28.431967 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.426268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.8.43.190:34469)
0925 15:15:28.426783 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.426793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.426825 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:15:28.426935 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.426942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.426995 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.427140 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.431501 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.431505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.431505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.431510 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.431571 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.431585 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.431660 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.431728 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.431786 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":65,"threads_started":1}
I20260925 15:15:28.460737 13025 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:28.460939 13025 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:28.464943 13025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2: Generated new TSK 0
I20260925 15:15:28.465138 13025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:15:28.475502 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34469
--builtin_ntp_servers=127.8.43.148:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:15:28.555902 13039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:28.556118 13039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:28.556139 13039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:28.557334 13039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:28.557376 13039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:28.557402 13039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:28.559345 13039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:28.559595 13039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:28.559904 13039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:28.564823 13047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:28.564863 13044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:28.565019 13039 server_base.cc:1034] running on GCE node
W20260925 15:15:28.565079 13045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:28.565306 13039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:28.565547 13039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:28.566715 13039 hybrid_clock.cc:648] HybridClock initialized: now 1790349328566698 us; error 45 us; skew 500 ppm
I20260925 15:15:28.567976 13039 webserver.cc:460] Webserver started at http://127.8.43.129:39333/ using document root <none> and password file <none>
I20260925 15:15:28.568228 13039 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:28.568297 13039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:28.568442 13039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:28.569491 13039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "962e128fa8d0472291a6c9ca38733ad6"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "51fbf6b27d6ad039dd8a2cf39a71ca33"
server_key_iv: "7a157a230b7fc8780181661d421d4b2b"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.569839 13039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "962e128fa8d0472291a6c9ca38733ad6"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "51fbf6b27d6ad039dd8a2cf39a71ca33"
server_key_iv: "7a157a230b7fc8780181661d421d4b2b"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.571127 13039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:28.571864 13053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.572031 13039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:28.572142 13039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "962e128fa8d0472291a6c9ca38733ad6"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "51fbf6b27d6ad039dd8a2cf39a71ca33"
server_key_iv: "7a157a230b7fc8780181661d421d4b2b"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.572219 13039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:28.575532 13039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:28.575789 13039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:28.575928 13039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:28.576133 13039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:28.576418 13039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:28.576468 13039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.576505 13039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:28.576531 13039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.582175 13039 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:36613
I20260925 15:15:28.582212 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:36613 every 8 connection(s)
I20260925 15:15:28.582532 13039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:28.585487 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.583258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:46241 (local address 127.8.43.190:34469)
0925 15:15:28.583325 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.583328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.583536 (+ 208us) server_negotiation.cc:407] Connection header received
0925 15:15:28.583966 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.583968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.583975 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.584031 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.585237 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.585238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.585238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.585242 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.585258 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.585263 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.585328 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.585417 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.585434 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:15:28.585640 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.583252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34469 (local address 127.8.43.129:46241)
0925 15:15:28.583416 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.583449 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.583888 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.584090 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.584094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.584101 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.585081 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:15:28.585082 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.585109 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:15:28.585137 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.585328 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.585332 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.585334 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.585392 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.585440 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260925 15:15:28.586071 13167 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34469
I20260925 15:15:28.586193 13167 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:28.586335 13167 heartbeater.cc:511] Master 127.8.43.190:34469 requested a full tablet report, sending...
I20260925 15:15:28.586817 12979 ts_manager.cc:194] Registered new tserver with Master: 962e128fa8d0472291a6c9ca38733ad6 (127.8.43.129:36613)
I20260925 15:15:28.590327 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13039
I20260925 15:15:28.590415 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:28.590612 8366 external_mini_cluster.cc:1428] Setting key 7bd1dc985740fa13f7a006d9b05be019
I20260925 15:15:28.591552 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34469
--builtin_ntp_servers=127.8.43.148:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:15:28.672118 13170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:28.672314 13170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:28.672335 13170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:28.673494 13170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:28.673535 13170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:28.673561 13170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:28.674985 13170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:28.675249 13170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:28.675529 13170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:28.680545 13178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:28.680490 13176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:28.680501 13175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:28.680949 13170 server_base.cc:1034] running on GCE node
I20260925 15:15:28.681167 13170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:28.681397 13170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:28.682592 13170 hybrid_clock.cc:648] HybridClock initialized: now 1790349328682581 us; error 33 us; skew 500 ppm
I20260925 15:15:28.683895 13170 webserver.cc:460] Webserver started at http://127.8.43.130:36305/ using document root <none> and password file <none>
I20260925 15:15:28.684158 13170 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:28.684232 13170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:28.684376 13170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:28.685510 13170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "b4cb6e7c188c4fb99a6e48402223ce13"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "d61fce8a40b68a0a4c78f0c41e09144f"
server_key_iv: "9f5f21e9edf891edbb6d419baa8b225f"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.685906 13170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "b4cb6e7c188c4fb99a6e48402223ce13"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "d61fce8a40b68a0a4c78f0c41e09144f"
server_key_iv: "9f5f21e9edf891edbb6d419baa8b225f"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.687175 13170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:28.688057 13184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.688238 13170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:28.688339 13170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "b4cb6e7c188c4fb99a6e48402223ce13"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "d61fce8a40b68a0a4c78f0c41e09144f"
server_key_iv: "9f5f21e9edf891edbb6d419baa8b225f"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.688455 13170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:28.691928 13170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:28.692181 13170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:28.692325 13170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:28.692548 13170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:28.692843 13170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:28.692894 13170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.692930 13170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:28.692960 13170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.699364 13170 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:41287
I20260925 15:15:28.699404 13297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:41287 every 8 connection(s)
I20260925 15:15:28.699759 13170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:28.702862 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.700711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:41587 (local address 127.8.43.190:34469)
0925 15:15:28.700779 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.700781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.700902 (+ 121us) server_negotiation.cc:407] Connection header received
0925 15:15:28.701348 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.701349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.701355 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.701471 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.702655 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.702657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.702657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.702661 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.702677 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.702681 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.702743 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.702791 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.702808 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:15:28.703038 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.700582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34469 (local address 127.8.43.130:41587)
0925 15:15:28.700768 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.700808 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.701260 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.701560 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.701564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.701572 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.702501 (+ 929us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:15:28.702502 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.702531 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:15:28.702558 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.702742 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.702745 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.702747 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.702797 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.702844 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260925 15:15:28.703575 13298 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34469
I20260925 15:15:28.703685 13298 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:28.703838 13298 heartbeater.cc:511] Master 127.8.43.190:34469 requested a full tablet report, sending...
I20260925 15:15:28.704216 12979 ts_manager.cc:194] Registered new tserver with Master: b4cb6e7c188c4fb99a6e48402223ce13 (127.8.43.130:41287)
I20260925 15:15:28.706143 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13170
I20260925 15:15:28.706225 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:28.706423 8366 external_mini_cluster.cc:1428] Setting key fc35e4a06a9ca0206652daee34233e65
I20260925 15:15:28.707387 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34469
--builtin_ntp_servers=127.8.43.148:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:15:28.786891 13301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:28.787122 13301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:28.787155 13301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:28.788340 13301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:28.788408 13301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:28.788451 13301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:28.789769 13301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:28.790030 13301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:28.790313 13301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:28.795176 13307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:28.795415 13309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:28.795269 13306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:28.795544 13301 server_base.cc:1034] running on GCE node
I20260925 15:15:28.795722 13301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:28.795944 13301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:28.797091 13301 hybrid_clock.cc:648] HybridClock initialized: now 1790349328797065 us; error 39 us; skew 500 ppm
I20260925 15:15:28.798213 13301 webserver.cc:460] Webserver started at http://127.8.43.131:43289/ using document root <none> and password file <none>
I20260925 15:15:28.798395 13301 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:28.798435 13301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:28.798548 13301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:28.799599 13301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "e1251e99393d417cb31ed0ed312306ae"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "f508f7b01d8ae6a905e38dc078b8728c"
server_key_iv: "d2363eca2ed48d7dc170c2b5cec0be3e"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.799938 13301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "e1251e99393d417cb31ed0ed312306ae"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "f508f7b01d8ae6a905e38dc078b8728c"
server_key_iv: "d2363eca2ed48d7dc170c2b5cec0be3e"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.801107 13301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:28.801784 13315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.801940 13301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:28.802042 13301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "e1251e99393d417cb31ed0ed312306ae"
format_stamp: "Formatted at 2026-09-25 15:15:28 on dist-test-slave-c5d6"
server_key: "f508f7b01d8ae6a905e38dc078b8728c"
server_key_iv: "d2363eca2ed48d7dc170c2b5cec0be3e"
server_key_version: "encryptionkey@0"
I20260925 15:15:28.802114 13301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:28.805316 13301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:28.805589 13301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:28.805727 13301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:28.805931 13301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:28.806214 13301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:28.806264 13301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.806303 13301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:28.806329 13301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:28.812106 13301 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:33445
I20260925 15:15:28.812140 13428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:33445 every 8 connection(s)
I20260925 15:15:28.812435 13301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:28.815192 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.813227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:44783 (local address 127.8.43.190:34469)
0925 15:15:28.813283 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.813285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.813496 (+ 211us) server_negotiation.cc:407] Connection header received
0925 15:15:28.813906 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.813907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.813912 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.813977 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.815034 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.815035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.815035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.815038 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.815047 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.815050 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.815088 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.815132 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.815143 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:15:28.815423 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.813202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34469 (local address 127.8.43.131:44783)
0925 15:15:28.813361 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.813400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.813830 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.814029 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.814032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.814039 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.814916 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:15:28.814917 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.814943 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:15:28.814969 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.815087 (+ 118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.815089 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.815091 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.815138 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.815180 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260925 15:15:28.815960 13429 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34469
I20260925 15:15:28.816052 13429 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:28.816220 13429 heartbeater.cc:511] Master 127.8.43.190:34469 requested a full tablet report, sending...
I20260925 15:15:28.816551 12979 ts_manager.cc:194] Registered new tserver with Master: e1251e99393d417cb31ed0ed312306ae (127.8.43.131:33445)
I20260925 15:15:28.821846 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13301
I20260925 15:15:28.821934 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:28.822085 8366 external_mini_cluster.cc:1428] Setting key df22dd9a37a0cc832fc9a7ea529258a6
I20260925 15:15:28.823022 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:15:28.825474 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.824314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.8.43.190:34469)
0925 15:15:28.824352 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.824354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.824472 (+ 118us) server_negotiation.cc:407] Connection header received
0925 15:15:28.824560 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.824561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.824567 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.824624 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.825227 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.825228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.825228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.825232 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.825259 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.825263 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.825305 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.825366 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.825378 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 15:15:28.826905 12979 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:28.828498 12979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:28.832453 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.830989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.8.43.129:36613)
0925 15:15:28.831138 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.831143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.831348 (+ 205us) server_negotiation.cc:407] Connection header received
0925 15:15:28.831423 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.831426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.831437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.831541 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.832113 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.832117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.832118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.832122 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.832150 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.832171 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.832247 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.832355 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.832371 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260925 15:15:28.832674 13441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.830871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:36613 (local address 127.0.0.1:37134)
0925 15:15:28.831207 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.831270 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.831337 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.831546 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.831552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.831558 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.832027 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:15:28.832027 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.832041 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:15:28.832059 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.832242 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.832248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.832251 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.832587 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.832601 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":52,"threads_started":1}
I20260925 15:15:28.832999 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.831087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:41287 (local address 127.0.0.1:41538)
0925 15:15:28.831496 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.831506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.831573 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.831979 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.831980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.831983 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.832563 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:15:28.832564 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.832570 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:15:28.832582 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.832861 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.832862 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.832863 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.832913 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.832937 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":67,"threads_started":1}
I20260925 15:15:28.833163 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.831246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.8.43.130:41287)
0925 15:15:28.831439 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.831443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.831553 (+ 110us) server_negotiation.cc:407] Connection header received
0925 15:15:28.831640 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.831643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.831652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.831700 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.832681 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.832690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.832691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.832696 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.832729 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.832758 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.832804 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.832922 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.833018 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
I20260925 15:15:28.833241 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.830883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:33445 (local address 127.0.0.1:50348)
0925 15:15:28.831207 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.831270 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.831337 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.832379 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.832380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.832382 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.832839 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:15:28.832839 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.832845 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:15:28.832856 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.833133 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.833133 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.833134 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.833172 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.833184 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":90,"threads_started":1}
I20260925 15:15:28.833489 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.830988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.8.43.131:33445)
0925 15:15:28.831213 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.831233 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.831381 (+ 148us) server_negotiation.cc:407] Connection header received
0925 15:15:28.832266 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.832271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.832281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.832361 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.832926 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.832932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.832932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.832936 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.832967 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.832988 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.833039 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.833381 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.833397 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260925 15:15:28.833503 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 8afcf7805b874a17bc8509679b856376 (DEFAULT_TABLE table=test-table [id=8f3cfe2e8eda40639f655a6ee2891b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:28.833886 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afcf7805b874a17bc8509679b856376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:28.834164 13363 tablet_service.cc:1467] Processing CreateTablet for tablet 8afcf7805b874a17bc8509679b856376 (DEFAULT_TABLE table=test-table [id=8f3cfe2e8eda40639f655a6ee2891b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:28.834497 13363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afcf7805b874a17bc8509679b856376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:28.835810 13232 tablet_service.cc:1467] Processing CreateTablet for tablet 8afcf7805b874a17bc8509679b856376 (DEFAULT_TABLE table=test-table [id=8f3cfe2e8eda40639f655a6ee2891b0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:28.836123 13232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afcf7805b874a17bc8509679b856376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:28.837265 13443 tablet_bootstrap.cc:492] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6: Bootstrap starting.
I20260925 15:15:28.837693 13444 tablet_bootstrap.cc:492] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: Bootstrap starting.
I20260925 15:15:28.838117 13443 tablet_bootstrap.cc:654] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:28.838325 13444 tablet_bootstrap.cc:654] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:28.838462 13443 log.cc:824] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:28.838649 13444 log.cc:824] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:28.839291 13443 tablet_bootstrap.cc:492] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6: No bootstrap required, opened a new log
I20260925 15:15:28.839359 13443 ts_tablet_manager.cc:1397] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:15:28.839365 13447 tablet_bootstrap.cc:492] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13: Bootstrap starting.
I20260925 15:15:28.839633 13444 tablet_bootstrap.cc:492] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: No bootstrap required, opened a new log
I20260925 15:15:28.839728 13444 ts_tablet_manager.cc:1397] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:28.840013 13447 tablet_bootstrap.cc:654] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:28.840304 13447 log.cc:824] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:28.841068 13447 tablet_bootstrap.cc:492] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13: No bootstrap required, opened a new log
I20260925 15:15:28.841156 13447 ts_tablet_manager.cc:1397] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:28.841984 13443 raft_consensus.cc:348] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.842149 13443 raft_consensus.cc:374] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:28.842182 13443 raft_consensus.cc:729] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962e128fa8d0472291a6c9ca38733ad6, State: Initialized, Role: FOLLOWER
I20260925 15:15:28.842254 13444 raft_consensus.cc:348] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.842319 13443 consensus_queue.cc:260] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.842398 13444 raft_consensus.cc:374] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:28.842435 13444 raft_consensus.cc:729] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1251e99393d417cb31ed0ed312306ae, State: Initialized, Role: FOLLOWER
I20260925 15:15:28.842564 13443 ts_tablet_manager.cc:1428] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:28.842555 13444 consensus_queue.cc:260] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.842680 13167 heartbeater.cc:503] Master 127.8.43.190:34469 was elected leader, sending a full tablet report...
I20260925 15:15:28.842794 13444 ts_tablet_manager.cc:1428] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:15:28.842885 13429 heartbeater.cc:503] Master 127.8.43.190:34469 was elected leader, sending a full tablet report...
I20260925 15:15:28.843598 13447 raft_consensus.cc:348] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.843763 13447 raft_consensus.cc:374] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:28.843827 13447 raft_consensus.cc:729] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cb6e7c188c4fb99a6e48402223ce13, State: Initialized, Role: FOLLOWER
I20260925 15:15:28.843945 13447 consensus_queue.cc:260] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.844204 13447 ts_tablet_manager.cc:1428] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:28.844264 13298 heartbeater.cc:503] Master 127.8.43.190:34469 was elected leader, sending a full tablet report...
I20260925 15:15:28.945122 13450 raft_consensus.cc:482] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:28.945286 13450 raft_consensus.cc:504] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.945654 13450 leader_election.cc:288] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 962e128fa8d0472291a6c9ca38733ad6 (127.8.43.129:36613), b4cb6e7c188c4fb99a6e48402223ce13 (127.8.43.130:41287)
I20260925 15:15:28.947009 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.945863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:36613 (local address 127.8.43.131:33857)
0925 15:15:28.945932 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.945944 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.946020 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.946193 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.946194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.946197 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.946646 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:15:28.946647 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.946655 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:15:28.946670 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.946867 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.946867 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.946868 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.946922 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.946936 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:15:28.947024 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.945976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:33857 (local address 127.8.43.129:36613)
0925 15:15:28.946055 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.946057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.946068 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:15:28.946097 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.946098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.946104 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.946167 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.946761 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.946763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.946763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.946768 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.946786 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.946791 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.946838 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.946937 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.946951 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:15:28.947412 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afcf7805b874a17bc8509679b856376" candidate_uuid: "e1251e99393d417cb31ed0ed312306ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962e128fa8d0472291a6c9ca38733ad6" is_pre_election: true
I20260925 15:15:28.947624 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.946018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:41287 (local address 127.8.43.131:36299)
0925 15:15:28.946230 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.946240 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:28.946297 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:28.946715 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:28.946716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:28.946720 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:28.947254 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:15:28.947255 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:28.947261 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:15:28.947274 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:28.947482 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:28.947483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:28.947484 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:15:28.947536 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:15:28.947550 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":139,"threads_started":1}
I20260925 15:15:28.947614 13121 raft_consensus.cc:2454] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1251e99393d417cb31ed0ed312306ae in term 0.
I20260925 15:15:28.947638 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.946145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:36299 (local address 127.8.43.130:41287)
0925 15:15:28.946190 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.946192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.946304 (+ 112us) server_negotiation.cc:407] Connection header received
0925 15:15:28.946617 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.946619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.946623 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.946693 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.947383 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.947388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.947389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.947395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.947407 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.947415 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.947474 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.947561 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.947575 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:15:28.947871 13319 leader_election.cc:302] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962e128fa8d0472291a6c9ca38733ad6, e1251e99393d417cb31ed0ed312306ae; no voters:
I20260925 15:15:28.948035 13252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afcf7805b874a17bc8509679b856376" candidate_uuid: "e1251e99393d417cb31ed0ed312306ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" is_pre_election: true
I20260925 15:15:28.948174 13252 raft_consensus.cc:2454] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1251e99393d417cb31ed0ed312306ae in term 0.
I20260925 15:15:28.948288 13450 raft_consensus.cc:2781] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:28.948350 13450 raft_consensus.cc:482] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:28.948383 13450 raft_consensus.cc:3037] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:28.949088 13450 raft_consensus.cc:504] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.949220 13450 leader_election.cc:288] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [CANDIDATE]: Term 1 election: Requested vote from peers 962e128fa8d0472291a6c9ca38733ad6 (127.8.43.129:36613), b4cb6e7c188c4fb99a6e48402223ce13 (127.8.43.130:41287)
I20260925 15:15:28.949369 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afcf7805b874a17bc8509679b856376" candidate_uuid: "e1251e99393d417cb31ed0ed312306ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962e128fa8d0472291a6c9ca38733ad6"
I20260925 15:15:28.949400 13252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afcf7805b874a17bc8509679b856376" candidate_uuid: "e1251e99393d417cb31ed0ed312306ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4cb6e7c188c4fb99a6e48402223ce13"
I20260925 15:15:28.949442 13121 raft_consensus.cc:3037] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:28.949458 13252 raft_consensus.cc:3037] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:15:28.950285 13299 tablet.cc:2369] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:28.950320 13252 raft_consensus.cc:2454] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1251e99393d417cb31ed0ed312306ae in term 1.
I20260925 15:15:28.950320 13121 raft_consensus.cc:2454] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1251e99393d417cb31ed0ed312306ae in term 1.
I20260925 15:15:28.950459 13319 leader_election.cc:302] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962e128fa8d0472291a6c9ca38733ad6, e1251e99393d417cb31ed0ed312306ae; no voters:
I20260925 15:15:28.950543 13450 raft_consensus.cc:2781] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:28.950683 13450 raft_consensus.cc:686] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 1 LEADER]: Becoming Leader. State: Replica: e1251e99393d417cb31ed0ed312306ae, State: Running, Role: LEADER
I20260925 15:15:28.950778 13450 consensus_queue.cc:237] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } }
I20260925 15:15:28.951694 12979 catalog_manager.cc:5495] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae reported cstate change: term changed from 0 to 1, leader changed from <none> to e1251e99393d417cb31ed0ed312306ae (127.8.43.131). New cstate: current_term: 1 leader_uuid: "e1251e99393d417cb31ed0ed312306ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1251e99393d417cb31ed0ed312306ae" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 33445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 } health_report { overall_health: UNKNOWN } } }
I20260925 15:15:28.978910 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.978071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.8.43.131:33445)
0925 15:15:28.978121 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:28.978123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:28.978131 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:28.978167 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:28.978168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:28.978181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:28.978237 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:28.978729 (+ 492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:28.978730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:28.978730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:28.978734 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:28.978748 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:28.978752 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:28.978797 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:28.978842 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 15:15:28.978853 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:28.980826 13252 raft_consensus.cc:1261] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 1 FOLLOWER]: Refusing update from remote peer e1251e99393d417cb31ed0ed312306ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:28.980826 13121 raft_consensus.cc:1261] T 8afcf7805b874a17bc8509679b856376 P 962e128fa8d0472291a6c9ca38733ad6 [term 1 FOLLOWER]: Refusing update from remote peer e1251e99393d417cb31ed0ed312306ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:28.981195 13453 consensus_queue.cc:1035] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "962e128fa8d0472291a6c9ca38733ad6" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:28.981339 13450 consensus_queue.cc:1035] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4cb6e7c188c4fb99a6e48402223ce13" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:28.983625 13456 mvcc.cc:205] Tried to move back new op lower bound from 7333270851502456832 to 7333270851383152640. Current Snapshot: MvccSnapshot[applied={T|T < 7333270851502456832}]
I20260925 15:15:28.983987 13458 mvcc.cc:205] Tried to move back new op lower bound from 7333270851502456832 to 7333270851383152640. Current Snapshot: MvccSnapshot[applied={T|T < 7333270851502456832}]
I20260925 15:15:28.984337 13457 mvcc.cc:205] Tried to move back new op lower bound from 7333270851502456832 to 7333270851383152640. Current Snapshot: MvccSnapshot[applied={T|T < 7333270851502456832 or (T in {7333270851502456832})}]
I20260925 15:15:28.986523 12979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:28.986622 12979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:28.987392 12979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90da8c3e30b64915b4a604605b77b7e2: Sending DeleteTablet for 3 replicas of tablet 8afcf7805b874a17bc8509679b856376
I20260925 15:15:28.987706 13363 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afcf7805b874a17bc8509679b856376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:28 UTC) from {username='slave'} at 127.0.0.1:50348
I20260925 15:15:28.987759 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afcf7805b874a17bc8509679b856376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:28 UTC) from {username='slave'} at 127.0.0.1:37134
I20260925 15:15:28.987809 13232 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afcf7805b874a17bc8509679b856376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:28 UTC) from {username='slave'} at 127.0.0.1:41538
I20260925 15:15:28.988091 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13039
I20260925 15:15:28.989112 13466 tablet_replica.cc:330] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: stopping tablet replica
I20260925 15:15:28.989218 13466 raft_consensus.cc:2229] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:28.989348 13466 raft_consensus.cc:2258] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:28.989804 13466 ts_tablet_manager.cc:1905] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:28.991756 13466 ts_tablet_manager.cc:1918] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:28.991912 13466 log.cc:1196] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/8afcf7805b874a17bc8509679b856376
I20260925 15:15:28.992255 13466 ts_tablet_manager.cc:1939] T 8afcf7805b874a17bc8509679b856376 P e1251e99393d417cb31ed0ed312306ae: Deleting consensus metadata
I20260925 15:15:28.992753 12963 catalog_manager.cc:4841] TS e1251e99393d417cb31ed0ed312306ae (127.8.43.131:33445): tablet 8afcf7805b874a17bc8509679b856376 (table test-table [id=8f3cfe2e8eda40639f655a6ee2891b0e]) successfully deleted
I20260925 15:15:28.995366 13468 tablet_replica.cc:330] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13: stopping tablet replica
I20260925 15:15:28.995497 13468 raft_consensus.cc:2229] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:28.995622 13468 raft_consensus.cc:2258] T 8afcf7805b874a17bc8509679b856376 P b4cb6e7c188c4fb99a6e48402223ce13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:28.995903 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13170
W20260925 15:15:28.996856 12966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:36613 (error 108)
I20260925 15:15:28.997346 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:28.997195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:36613 (local address 127.0.0.1:37150)
0925 15:15:28.997257 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:28.997304 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:15:28.997506 12966 catalog_manager.cc:4584] TS 962e128fa8d0472291a6c9ca38733ad6 (127.8.43.129:36613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8afcf7805b874a17bc8509679b856376: Network error: Client connection negotiation failed: client connection to 127.8.43.129:36613: connect: Connection refused (error 111)
I20260925 15:15:29.001582 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13301
I20260925 15:15:29.001731 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.001629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:41287 (local address 127.0.0.1:41550)
0925 15:15:29.001678 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:29.001699 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.130:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 15:15:29.006121 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 12948
2026-09-25T15:15:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (721 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 15:15:29.025767 8366 mini_oidc.cc:152] Starting JWKS server
I20260925 15:15:29.026950 8366 webserver.cc:460] Webserver started at https://127.0.0.1:36131/ using document root <none> and password file <none>
I20260925 15:15:29.027084 8366 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:15:29.027314 8366 webserver.cc:460] Webserver started at http://127.0.0.1:37981/ using document root <none> and password file <none>
2026-09-25T15:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:29Z Disabled control of system clock
I20260925 15:15:29.035077 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:45189
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:40563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:45189
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36131/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/testchainca.cert with env {}
W20260925 15:15:29.113282 13475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:29.113543 13475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:29.113567 13475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:29.114703 13475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:29.114740 13475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:29.114753 13475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:29.114764 13475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:29.114776 13475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:15:29.114789 13475 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:15:29.114800 13475 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36131/jwks.json
I20260925 15:15:29.116276 13475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:40563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:45189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:45189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36131/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:29.116473 13475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:29.116629 13475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:29.121337 13483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:29.121562 13480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:29.121393 13481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:29.121944 13475 server_base.cc:1034] running on GCE node
I20260925 15:15:29.122228 13475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:29.122469 13475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:29.123631 13475 hybrid_clock.cc:648] HybridClock initialized: now 1790349329123607 us; error 40 us; skew 500 ppm
I20260925 15:15:29.124910 13475 webserver.cc:460] Webserver started at http://127.8.43.190:43609/ using document root <none> and password file <none>
I20260925 15:15:29.125155 13475 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:29.125226 13475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:29.125386 13475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:29.126524 13475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "07300d51ac5d4771a85dace7fabbab98"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "73622c944f538a09f1ecb9779d15c368"
server_key_iv: "0a4e30776f66038477416e0119091640"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.126899 13475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "07300d51ac5d4771a85dace7fabbab98"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "73622c944f538a09f1ecb9779d15c368"
server_key_iv: "0a4e30776f66038477416e0119091640"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.128268 13475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:15:29.128999 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:29.129155 13475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:29.129259 13475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "07300d51ac5d4771a85dace7fabbab98"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "73622c944f538a09f1ecb9779d15c368"
server_key_iv: "0a4e30776f66038477416e0119091640"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.129331 13475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:29.154469 13475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:29.154803 13475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:29.154966 13475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:29.159272 13475 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:45189
I20260925 15:15:29.159291 13541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:45189 every 8 connection(s)
I20260925 15:15:29.159659 13475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:29.160765 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:29.163205 13542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98: Bootstrap starting.
I20260925 15:15:29.163878 13542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:29.164186 13542 log.cc:824] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:29.164922 13542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98: No bootstrap required, opened a new log
I20260925 15:15:29.166584 13542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07300d51ac5d4771a85dace7fabbab98" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45189 } }
I20260925 15:15:29.166731 13542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:29.166754 13542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07300d51ac5d4771a85dace7fabbab98, State: Initialized, Role: FOLLOWER
I20260925 15:15:29.166918 13542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07300d51ac5d4771a85dace7fabbab98" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45189 } }
I20260925 15:15:29.166981 13542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:29.167002 13542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:29.167071 13542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:29.167645 13542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07300d51ac5d4771a85dace7fabbab98" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45189 } }
I20260925 15:15:29.167763 13542 leader_election.cc:302] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07300d51ac5d4771a85dace7fabbab98; no voters:
I20260925 15:15:29.167927 13542 leader_election.cc:288] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:29.168004 13545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:29.168148 13545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [term 1 LEADER]: Becoming Leader. State: Replica: 07300d51ac5d4771a85dace7fabbab98, State: Running, Role: LEADER
I20260925 15:15:29.168211 13542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:29.168242 13545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07300d51ac5d4771a85dace7fabbab98" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45189 } }
I20260925 15:15:29.168666 13547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07300d51ac5d4771a85dace7fabbab98. Latest consensus state: current_term: 1 leader_uuid: "07300d51ac5d4771a85dace7fabbab98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07300d51ac5d4771a85dace7fabbab98" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45189 } } }
I20260925 15:15:29.168744 13546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07300d51ac5d4771a85dace7fabbab98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07300d51ac5d4771a85dace7fabbab98" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45189 } } }
I20260925 15:15:29.168751 13547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:29.168859 13546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:29.169055 13554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:29.169169 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13475
I20260925 15:15:29.169274 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:29.169483 8366 external_mini_cluster.cc:1428] Setting key 594806be6579a023dbc6935db73fe942
I20260925 15:15:29.169572 13554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:29.171160 13554 catalog_manager.cc:1347] Generated new cluster ID: e266160d1ea34d50a29ffe15e06e0a3f
I20260925 15:15:29.171237 13554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:29.176623 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.169915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.8.43.190:45189)
0925 15:15:29.170863 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:29.170873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:29.170903 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:15:29.171007 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:29.171012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:29.171129 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:29.171581 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:29.171667 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.173144 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.174284 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.174911 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.175033 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.175818 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:29.175822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:29.175822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:29.176031 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:29.176069 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:29.176083 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:29.176145 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:29.176237 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:15:29.176409 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":78,"threads_started":1}
I20260925 15:15:29.177510 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:15:29.188654 13554 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:29.189160 13554 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:29.194857 13554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07300d51ac5d4771a85dace7fabbab98: Generated new TSK 0
I20260925 15:15:29.195045 13554 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:15:29.222901 13572 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 15:15:29.223080 13572 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:15:29.223409 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.206616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.8.43.190:45189)
0925 15:15:29.206674 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:29.206676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:29.206813 (+ 137us) server_negotiation.cc:407] Connection header received
0925 15:15:29.206881 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:29.206882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:29.206906 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:29.206991 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:15:29.207166 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.207991 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.208773 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.209105 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.209174 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.209347 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:15:29.223145 (+ 13798us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:15:29.223336 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 15:15:29.223469 13558 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 15:15:29.223773 13571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:29.206527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:45189 (local address 127.0.0.1:32772)
0925 15:15:29.206750 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:29.206759 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:29.206813 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:29.207025 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:29.207026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:29.207028 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:15:29.207092 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:29.207095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.208091 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.208092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:29.208695 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:29.208697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.209181 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.209181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:29.209225 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.209243 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:15:29.223310 (+ 14067us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:15:29.223414 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.43.190:45189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.43.190:45189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 15:15:29.224313 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13475
2026-09-25T15:15:29Z chronyd exiting
I20260925 15:15:29.259648 8366 test_util.cc:175] -----------------------------------------------
I20260925 15:15:29.259752 8366 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349306597827-8366-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (235 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 15:15:29.260500 8366 mini_oidc.cc:152] Starting JWKS server
I20260925 15:15:29.261197 8366 webserver.cc:460] Webserver started at https://127.0.0.1:46629/ using document root <none> and password file <none>
I20260925 15:15:29.261265 8366 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:15:29.261430 8366 webserver.cc:460] Webserver started at http://127.0.0.1:42957/ using document root <none> and password file <none>
2026-09-25T15:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:29Z Disabled control of system clock
I20260925 15:15:29.269371 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:43125
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:43125
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46629/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/testchainca.cert with env {}
W20260925 15:15:29.352327 13578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:29.352522 13578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:29.352545 13578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:29.353739 13578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:29.353776 13578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:29.353788 13578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:29.353832 13578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:29.353856 13578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:15:29.353881 13578 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:15:29.353928 13578 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46629/jwks.json
I20260925 15:15:29.355455 13578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:43125
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:43125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46629/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:29.355684 13578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:29.355894 13578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:29.360539 13584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:29.360563 13583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:29.360577 13586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:29.360608 13578 server_base.cc:1034] running on GCE node
I20260925 15:15:29.360986 13578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:29.361272 13578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:29.362448 13578 hybrid_clock.cc:648] HybridClock initialized: now 1790349329362423 us; error 52 us; skew 500 ppm
I20260925 15:15:29.363742 13578 webserver.cc:460] Webserver started at http://127.8.43.190:36699/ using document root <none> and password file <none>
I20260925 15:15:29.364001 13578 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:29.364071 13578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:29.364215 13578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:29.365218 13578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "f28c3309001a435ea53ed041564514fc"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "39839c86d9a07db1061abdbc0ff91c10"
server_key_iv: "5f731ba222c6d69d9a06766763d4633d"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.365599 13578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "f28c3309001a435ea53ed041564514fc"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "39839c86d9a07db1061abdbc0ff91c10"
server_key_iv: "5f731ba222c6d69d9a06766763d4633d"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.366990 13578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:29.367749 13592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:29.367930 13578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:29.368024 13578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "f28c3309001a435ea53ed041564514fc"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "39839c86d9a07db1061abdbc0ff91c10"
server_key_iv: "5f731ba222c6d69d9a06766763d4633d"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.368106 13578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:29.386307 13578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:29.386618 13578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:29.386775 13578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:29.390568 13578 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:43125
I20260925 15:15:29.390606 13644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:43125 every 8 connection(s)
I20260925 15:15:29.390978 13578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:29.391988 13645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:29.393657 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13578
I20260925 15:15:29.393774 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:29.393986 8366 external_mini_cluster.cc:1428] Setting key 13a9b6acf38a579b2c30979625d3363a
I20260925 15:15:29.394750 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc: Bootstrap starting.
I20260925 15:15:29.395406 13645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:29.395700 13645 log.cc:824] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:29.396412 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc: No bootstrap required, opened a new log
I20260925 15:15:29.398231 13645 raft_consensus.cc:348] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28c3309001a435ea53ed041564514fc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43125 } }
I20260925 15:15:29.398419 13645 raft_consensus.cc:374] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:29.398460 13645 raft_consensus.cc:729] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28c3309001a435ea53ed041564514fc, State: Initialized, Role: FOLLOWER
I20260925 15:15:29.398587 13645 consensus_queue.cc:260] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28c3309001a435ea53ed041564514fc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43125 } }
I20260925 15:15:29.398679 13645 raft_consensus.cc:388] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:29.398742 13645 raft_consensus.cc:482] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:29.398804 13645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:29.399508 13645 raft_consensus.cc:504] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28c3309001a435ea53ed041564514fc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43125 } }
I20260925 15:15:29.399559 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.394525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.8.43.190:43125)
0925 15:15:29.394745 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:29.394754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:29.394787 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:15:29.395314 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:29.395329 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:29.395402 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:29.395562 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:29.395790 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.396742 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.397563 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.397972 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.398100 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.398865 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:29.398868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:29.398869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:29.399064 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:29.399098 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:29.399109 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:29.399187 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:29.399300 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:15:29.399382 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260925 15:15:29.399654 13645 leader_election.cc:302] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f28c3309001a435ea53ed041564514fc; no voters:
I20260925 15:15:29.399842 13645 leader_election.cc:288] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:29.400046 13651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:29.400107 13645 sys_catalog.cc:564] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:29.400429 13651 raft_consensus.cc:686] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [term 1 LEADER]: Becoming Leader. State: Replica: f28c3309001a435ea53ed041564514fc, State: Running, Role: LEADER
I20260925 15:15:29.400534 13651 consensus_queue.cc:237] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28c3309001a435ea53ed041564514fc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43125 } }
I20260925 15:15:29.400550 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:15:29.400859 13651 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f28c3309001a435ea53ed041564514fc. Latest consensus state: current_term: 1 leader_uuid: "f28c3309001a435ea53ed041564514fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28c3309001a435ea53ed041564514fc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43125 } } }
I20260925 15:15:29.400931 13651 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:29.401607 13650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f28c3309001a435ea53ed041564514fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28c3309001a435ea53ed041564514fc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 43125 } } }
I20260925 15:15:29.401916 13650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc [sys.catalog]: This master's current role is: LEADER
W20260925 15:15:29.402024 13666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:15:29.402072 13666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:15:29.402132 13659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:29.402311 13659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:29.403803 13659 catalog_manager.cc:1347] Generated new cluster ID: 34614e5d44ea46f3be44b7ab59d10bf3
I20260925 15:15:29.403858 13659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:29.417834 13659 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:29.418558 13659 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:29.431541 13659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f28c3309001a435ea53ed041564514fc: Generated new TSK 0
I20260925 15:15:29.431759 13659 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:15:29.450235 13675 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 15:15:29.450309 13675 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:15:29.450696 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.436398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.8.43.190:43125)
0925 15:15:29.436444 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:29.436446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:29.436550 (+ 104us) server_negotiation.cc:407] Connection header received
0925 15:15:29.436621 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:29.436622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:29.436632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:29.436691 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:15:29.436842 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.437347 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.438456 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.438676 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.438733 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.438918 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:15:29.450451 (+ 11533us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:15:29.450617 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 15:15:29.450752 13648 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 15:15:29.450778 13674 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:29.436302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:43125 (local address 127.0.0.1:51894)
0925 15:15:29.436473 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:29.436485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:29.436553 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:29.436714 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:29.436715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:29.436717 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:15:29.436781 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:29.436783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.437433 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.437434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:29.438365 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:29.438368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.438758 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.438758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:29.438818 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.438823 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:15:29.450568 (+ 11745us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:15:29.450675 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.43.190:43125: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260925 15:15:29.451061 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13578
2026-09-25T15:15:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 15:15:29.488397 8366 mini_oidc.cc:152] Starting JWKS server
I20260925 15:15:29.489085 8366 webserver.cc:460] Webserver started at https://127.0.0.1:42095/ using document root <none> and password file <none>
I20260925 15:15:29.489153 8366 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:15:29.489344 8366 webserver.cc:460] Webserver started at http://127.0.0.1:38929/ using document root <none> and password file <none>
2026-09-25T15:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:29Z Disabled control of system clock
I20260925 15:15:29.497051 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:45647
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:41633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:45647
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42095/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:15:29.575905 13681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:29.576099 13681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:29.576119 13681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:29.577224 13681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:29.577260 13681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:29.577271 13681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:29.577284 13681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:29.577296 13681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:15:29.577308 13681 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:15:29.577320 13681 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42095/jwks.json
I20260925 15:15:29.578598 13681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:45647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42095/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:29.578789 13681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:29.579015 13681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:29.583612 13689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:29.583595 13686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:29.583681 13687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:29.583791 13681 server_base.cc:1034] running on GCE node
I20260925 15:15:29.584102 13681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:29.584437 13681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:29.585608 13681 hybrid_clock.cc:648] HybridClock initialized: now 1790349329585584 us; error 35 us; skew 500 ppm
I20260925 15:15:29.586930 13681 webserver.cc:460] Webserver started at http://127.8.43.190:33851/ using document root <none> and password file <none>
I20260925 15:15:29.587195 13681 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:29.587330 13681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:29.587527 13681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:29.588572 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "55def38bfc7c4423b4649024c598d3e7"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "367a99f96f00926769ab74564a931fec"
server_key_iv: "019f1ed0455e24ab1a9eaedc44840f23"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.588941 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "55def38bfc7c4423b4649024c598d3e7"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "367a99f96f00926769ab74564a931fec"
server_key_iv: "019f1ed0455e24ab1a9eaedc44840f23"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.590263 13681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:29.591077 13695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:29.591288 13681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:29.591387 13681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "55def38bfc7c4423b4649024c598d3e7"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "367a99f96f00926769ab74564a931fec"
server_key_iv: "019f1ed0455e24ab1a9eaedc44840f23"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.591504 13681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:29.612335 13681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:29.612707 13681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:29.612886 13681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:29.617229 13681 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:45647
I20260925 15:15:29.617262 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:45647 every 8 connection(s)
I20260925 15:15:29.617645 13681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:29.618297 13748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:29.621286 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13681
I20260925 15:15:29.621315 13748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7: Bootstrap starting.
I20260925 15:15:29.621394 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:29.621579 8366 external_mini_cluster.cc:1428] Setting key 1c50b3d3452ab84d43815e7c60b935c6
I20260925 15:15:29.622056 13748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:29.622444 13748 log.cc:824] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:29.623252 13748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7: No bootstrap required, opened a new log
I20260925 15:15:29.625185 13748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def38bfc7c4423b4649024c598d3e7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45647 } }
I20260925 15:15:29.625346 13748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:29.625379 13748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55def38bfc7c4423b4649024c598d3e7, State: Initialized, Role: FOLLOWER
I20260925 15:15:29.625535 13748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def38bfc7c4423b4649024c598d3e7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45647 } }
I20260925 15:15:29.625622 13748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:29.625648 13748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:29.625713 13748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:29.626284 13748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def38bfc7c4423b4649024c598d3e7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45647 } }
I20260925 15:15:29.626399 13748 leader_election.cc:302] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55def38bfc7c4423b4649024c598d3e7; no voters:
I20260925 15:15:29.626567 13748 leader_election.cc:288] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:29.626749 13754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:29.626845 13748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:29.626977 13754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [term 1 LEADER]: Becoming Leader. State: Replica: 55def38bfc7c4423b4649024c598d3e7, State: Running, Role: LEADER
I20260925 15:15:29.627660 13754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def38bfc7c4423b4649024c598d3e7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45647 } }
I20260925 15:15:29.628010 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.622120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.8.43.190:45647)
0925 15:15:29.622308 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:29.622324 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:29.622352 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:15:29.623035 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:29.623043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:29.623106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:29.623279 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:29.623501 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.624405 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.625551 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.626081 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.626182 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.627147 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:29.627151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:29.627151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:29.627390 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:29.627431 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:29.627484 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:29.627590 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:29.627678 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:15:29.627760 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260925 15:15:29.628458 13755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55def38bfc7c4423b4649024c598d3e7. Latest consensus state: current_term: 1 leader_uuid: "55def38bfc7c4423b4649024c598d3e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def38bfc7c4423b4649024c598d3e7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45647 } } }
I20260925 15:15:29.628580 13753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55def38bfc7c4423b4649024c598d3e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55def38bfc7c4423b4649024c598d3e7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45647 } } }
I20260925 15:15:29.628690 13755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:29.628695 13753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:29.628924 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 15:15:29.629235 13767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:15:29.629328 13767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:15:29.629426 13768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:29.629592 13768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:29.631114 13768 catalog_manager.cc:1347] Generated new cluster ID: d2389e59254c4e55b9efce8030e85d4a
I20260925 15:15:29.631165 13768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:29.658231 13768 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:29.658785 13768 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:29.668761 13768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55def38bfc7c4423b4649024c598d3e7: Generated new TSK 0
I20260925 15:15:29.668978 13768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:15:29.689317 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.685613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.8.43.190:45647)
0925 15:15:29.685682 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:29.685684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:29.685815 (+ 131us) server_negotiation.cc:407] Connection header received
0925 15:15:29.685892 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:29.685893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:29.685903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:29.685964 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:15:29.686158 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.686629 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.687794 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.687983 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.688041 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.688225 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:15:29.689119 (+ 894us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:15:29.689245 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 15:15:29.689378 13751 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 15:15:29.689433 13777 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:29.685544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:45647 (local address 127.0.0.1:39200)
0925 15:15:29.685738 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:29.685750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:29.685822 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:29.685996 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:29.685997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:29.685999 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:15:29.686057 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:29.686059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.686730 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.686732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:29.687713 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:29.687717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.688066 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.688067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:29.688128 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.688133 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:15:29.689224 (+ 1091us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:15:29.689323 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.43.190:45647: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42095/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260925 15:15:29.691280 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13681
E20260925 15:15:29.700309 13778 webserver.cc:533] Webserver: sslize failed:
E20260925 15:15:29.700416 13778 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T15:15:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T15:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:29Z Disabled control of system clock
I20260925 15:15:29.736673 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:37147
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:37147
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/testchainca.cert with env {}
W20260925 15:15:29.815759 13782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:29.815970 13782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:29.815992 13782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:29.817240 13782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:29.817289 13782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:29.817302 13782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:29.817315 13782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:29.817369 13782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:15:29.817386 13782 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:15:29.817409 13782 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 15:15:29.818922 13782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:37147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:37147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:29.819182 13782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:29.819437 13782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:29.824159 13787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:29.824169 13790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:29.824244 13788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:29.824501 13782 server_base.cc:1034] running on GCE node
I20260925 15:15:29.824739 13782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:29.825045 13782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:29.826195 13782 hybrid_clock.cc:648] HybridClock initialized: now 1790349329826174 us; error 38 us; skew 500 ppm
I20260925 15:15:29.827559 13782 webserver.cc:460] Webserver started at http://127.8.43.190:40427/ using document root <none> and password file <none>
I20260925 15:15:29.827855 13782 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:29.827907 13782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:29.828092 13782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:29.829263 13782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "22c04b61bc0f6b52795983478c8d1061"
server_key_iv: "cd9079940e892b1e3dc2a951a4f8dc1b"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.829650 13782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "22c04b61bc0f6b52795983478c8d1061"
server_key_iv: "cd9079940e892b1e3dc2a951a4f8dc1b"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.831096 13782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:29.831864 13796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:29.832027 13782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:29.832141 13782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0"
format_stamp: "Formatted at 2026-09-25 15:15:29 on dist-test-slave-c5d6"
server_key: "22c04b61bc0f6b52795983478c8d1061"
server_key_iv: "cd9079940e892b1e3dc2a951a4f8dc1b"
server_key_version: "encryptionkey@0"
I20260925 15:15:29.832221 13782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:29.855805 13782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:29.856132 13782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:29.856297 13782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:29.860375 13782 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:37147
I20260925 15:15:29.860435 13848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:37147 every 8 connection(s)
I20260925 15:15:29.860858 13782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:29.861815 13849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:29.864212 13849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0: Bootstrap starting.
I20260925 15:15:29.864759 13849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:29.864987 13849 log.cc:824] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:29.865640 13849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0: No bootstrap required, opened a new log
I20260925 15:15:29.867138 13849 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37147 } }
I20260925 15:15:29.867336 13849 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:29.867363 13849 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fdbd57f51643c9bb5b56ba12eb3bc0, State: Initialized, Role: FOLLOWER
I20260925 15:15:29.867498 13849 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37147 } }
I20260925 15:15:29.867578 13849 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:29.867599 13849 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:29.867627 13849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:29.868093 13849 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37147 } }
I20260925 15:15:29.868171 13849 leader_election.cc:302] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9fdbd57f51643c9bb5b56ba12eb3bc0; no voters:
I20260925 15:15:29.868294 13849 leader_election.cc:288] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:29.868371 13852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:29.868517 13852 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [term 1 LEADER]: Becoming Leader. State: Replica: c9fdbd57f51643c9bb5b56ba12eb3bc0, State: Running, Role: LEADER
I20260925 15:15:29.868580 13849 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:29.868618 13852 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37147 } }
I20260925 15:15:29.869071 13854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9fdbd57f51643c9bb5b56ba12eb3bc0. Latest consensus state: current_term: 1 leader_uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37147 } } }
I20260925 15:15:29.869206 13853 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fdbd57f51643c9bb5b56ba12eb3bc0" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37147 } } }
I20260925 15:15:29.869310 13853 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:29.869220 13854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:29.869699 13861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:29.870143 13861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:29.870945 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13782
I20260925 15:15:29.871079 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:29.871331 8366 external_mini_cluster.cc:1428] Setting key 08ea614b962541785373a96da6a73a4b
I20260925 15:15:29.871903 13861 catalog_manager.cc:1347] Generated new cluster ID: 04c8b9ffb0f240fd82a4e40caf56dcf6
I20260925 15:15:29.871956 13861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:29.882221 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.871809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.8.43.190:37147)
0925 15:15:29.872212 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:29.872219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:29.872243 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:15:29.872341 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:29.872347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:29.872400 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:29.872549 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:29.872761 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.873720 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.874603 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.875070 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.881001 (+ 5931us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.881164 (+ 163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:29.881167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:29.881168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:15:29.881408 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:29.881452 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:15:29.881479 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:29.881554 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:29.881845 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 15:15:29.881940 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":77,"threads_started":1}
I20260925 15:15:29.882933 13861 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:29.883683 13861 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:29.887099 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:15:29.900684 13861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9fdbd57f51643c9bb5b56ba12eb3bc0: Generated new TSK 0
I20260925 15:15:29.900938 13861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:15:29.931842 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:29.909441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.8.43.190:37147)
0925 15:15:29.909506 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:29.909508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:29.909635 (+ 127us) server_negotiation.cc:407] Connection header received
0925 15:15:29.909709 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:29.909710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:29.909719 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:29.909779 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:15:29.909918 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.910419 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.911560 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.911758 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.911811 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.911986 (+ 175us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:15:29.931618 (+ 19632us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:15:29.931784 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48584: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 15:15:29.931929 13872 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48584: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 15:15:29.932031 13878 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:29.909379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:37147 (local address 127.0.0.1:48584)
0925 15:15:29.909562 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:29.909573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:29.909640 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:29.909795 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:29.909796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:29.909798 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:15:29.909855 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:29.909857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.910494 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.910495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:29.911485 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:29.911488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:29.911830 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:29.911831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:29.911890 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:29.911896 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:15:29.931779 (+ 19883us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:15:29.931913 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.43.190:37147: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
I20260925 15:15:29.932346 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13782
2026-09-25T15:15:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (220 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:30 dist-test-slave-c5d6 krb5kdc[13890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:30 dist-test-slave-c5d6 krb5kdc[13890](info): set up 2 sockets
Sep 25 15:15:30 dist-test-slave-c5d6 krb5kdc[13890](info): commencing operation
krb5kdc: starting...
W20260925 15:15:31.139436 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:15:31.252440 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:46209
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:46209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:31.332640 13903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:31.332835 13903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:31.332857 13903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:31.334022 13903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:31.334064 13903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:31.334079 13903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:31.334118 13903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:31.334147 13903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:31.335584 13903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:46209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:46209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:31.335827 13903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:31.336050 13903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:31.340655 13909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:31.340665 13911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:31.340796 13908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:31.340931 13903 server_base.cc:1034] running on GCE node
I20260925 15:15:31.341245 13903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:31.341517 13903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:31.342701 13903 hybrid_clock.cc:648] HybridClock initialized: now 1790349331342693 us; error 32 us; skew 500 ppm
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.350329 13903 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:15:31.350767 13903 webserver.cc:460] Webserver started at http://127.8.43.190:36765/ using document root <none> and password file <none>
I20260925 15:15:31.351017 13903 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:31.351086 13903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:31.351270 13903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:31.352336 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "082020d8ba6e47449a8a7f3bd65d1d3c"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "9c2f0f3f70aea0a70a60f29138d7b137"
server_key_iv: "3f9045f1f455c921ebedc44df3a922d0"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.352699 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "082020d8ba6e47449a8a7f3bd65d1d3c"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "9c2f0f3f70aea0a70a60f29138d7b137"
server_key_iv: "3f9045f1f455c921ebedc44df3a922d0"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.354014 13903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:15:31.354710 13918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.354877 13903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.354987 13903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "082020d8ba6e47449a8a7f3bd65d1d3c"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "9c2f0f3f70aea0a70a60f29138d7b137"
server_key_iv: "3f9045f1f455c921ebedc44df3a922d0"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.355068 13903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:31.379104 13903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:31.381420 13903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:31.381592 13903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:31.385635 13903 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:46209
I20260925 15:15:31.385689 13970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:46209 every 8 connection(s)
I20260925 15:15:31.386065 13903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:31.386965 13971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:31.387364 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13903
I20260925 15:15:31.387477 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:31.387744 8366 external_mini_cluster.cc:1428] Setting key b60525155a848a8d204ad8bb12fd9b1d
I20260925 15:15:31.390010 13971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c: Bootstrap starting.
I20260925 15:15:31.390705 13971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:31.390988 13971 log.cc:824] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:31.391726 13971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c: No bootstrap required, opened a new log
I20260925 15:15:31.393780 13971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082020d8ba6e47449a8a7f3bd65d1d3c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46209 } }
I20260925 15:15:31.393908 13971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:31.393929 13971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082020d8ba6e47449a8a7f3bd65d1d3c, State: Initialized, Role: FOLLOWER
I20260925 15:15:31.394059 13971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082020d8ba6e47449a8a7f3bd65d1d3c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46209 } }
I20260925 15:15:31.394125 13971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:31.394145 13971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:31.394209 13971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:31.394719 13971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082020d8ba6e47449a8a7f3bd65d1d3c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46209 } }
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:31.394800 13971 leader_election.cc:302] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 082020d8ba6e47449a8a7f3bd65d1d3c; no voters:
I20260925 15:15:31.394992 13971 leader_election.cc:288] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:31.395030 13976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:31.395150 13976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [term 1 LEADER]: Becoming Leader. State: Replica: 082020d8ba6e47449a8a7f3bd65d1d3c, State: Running, Role: LEADER
I20260925 15:15:31.395267 13971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:31.395314 13976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082020d8ba6e47449a8a7f3bd65d1d3c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46209 } }
I20260925 15:15:31.395651 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082020d8ba6e47449a8a7f3bd65d1d3c. Latest consensus state: current_term: 1 leader_uuid: "082020d8ba6e47449a8a7f3bd65d1d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082020d8ba6e47449a8a7f3bd65d1d3c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46209 } } }
I20260925 15:15:31.395726 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:31.395780 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "082020d8ba6e47449a8a7f3bd65d1d3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082020d8ba6e47449a8a7f3bd65d1d3c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46209 } } }
I20260925 15:15:31.395866 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c [sys.catalog]: This master's current role is: LEADER
W20260925 15:15:31.397254 13990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:15:31.397308 13990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:15:31.397369 13992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:31.397524 13992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:31.399039 13992 catalog_manager.cc:1347] Generated new cluster ID: 4edce8ad8aeb49deb72441cba8c7f2c0
I20260925 15:15:31.399087 13992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:31.399277 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.388391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.8.43.190:46209)
0925 15:15:31.388572 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.388582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.388618 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:15:31.389140 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.389149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.389209 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.389364 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.390043 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.391457 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.392205 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.392814 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.392920 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.395349 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:31.395354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:31.395354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:31.395365 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:31.397682 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.398010 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.398013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.398014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.398124 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.398368 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.398369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.398369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.398669 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:31.398801 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:31.398928 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:15:31.399066 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260925 15:15:31.408043 13992 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:31.408758 13992 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:31.429034 13992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 082020d8ba6e47449a8a7f3bd65d1d3c: Generated new TSK 0
I20260925 15:15:31.429267 13992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:15:31.453953 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46209
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:31.534695 13997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:31.534899 13997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:31.534921 13997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:31.536074 13997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:31.536115 13997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:31.536140 13997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:31.537663 13997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:31.537895 13997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:31.538072 13997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:15:31.543018 13997 server_base.cc:1034] running on GCE node
W20260925 15:15:31.542977 14003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:31.542958 14002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:31.542974 14005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:31.543460 13997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:31.543702 13997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:31.544862 13997 hybrid_clock.cc:648] HybridClock initialized: now 1790349331544848 us; error 42 us; skew 500 ppm
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.547369 13997 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:15:31.547747 13997 webserver.cc:460] Webserver started at http://127.8.43.129:45975/ using document root <none> and password file <none>
I20260925 15:15:31.547999 13997 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:31.548043 13997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:31.548130 13997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:31.549155 13997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "7de6efcc5daf4e9dbf8117fb7c4b0bbc"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "ed46e0548305c0ba074804faa1477bb9"
server_key_iv: "98629916d8bbf4ef32630de0dc6385f7"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.549518 13997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "7de6efcc5daf4e9dbf8117fb7c4b0bbc"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "ed46e0548305c0ba074804faa1477bb9"
server_key_iv: "98629916d8bbf4ef32630de0dc6385f7"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.550791 13997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:15:31.551621 14012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.551802 13997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:31.551904 13997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "7de6efcc5daf4e9dbf8117fb7c4b0bbc"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "ed46e0548305c0ba074804faa1477bb9"
server_key_iv: "98629916d8bbf4ef32630de0dc6385f7"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.551982 13997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:31.586997 13997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:31.589371 13997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:31.589496 13997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:31.589689 13997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:31.590032 13997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:31.590070 13997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.590093 13997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:31.590138 13997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.595780 13997 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:46855
I20260925 15:15:31.595898 14125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:46855 every 8 connection(s)
I20260925 15:15:31.596262 13997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:31.598627 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 13997
I20260925 15:15:31.598744 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:31.598917 8366 external_mini_cluster.cc:1428] Setting key c76cca7ea92fea902d622ed08b6d5193
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:31.606400 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.597044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:53429 (local address 127.8.43.190:46209)
0925 15:15:31.597110 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.597112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.597321 (+ 209us) server_negotiation.cc:407] Connection header received
0925 15:15:31.598238 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.598240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.598249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.598298 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.598976 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.599745 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.600767 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.601001 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.601084 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.603594 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:31.603596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:31.603596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:31.603605 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:31.604972 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.605403 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.605404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.605405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.605469 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.605743 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.605744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.605745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.605861 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:31.605937 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:31.606218 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:15:31.606308 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:15:31.606590 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.597029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46209 (local address 127.8.43.129:53429)
0925 15:15:31.597217 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:31.597253 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:31.598145 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:31.598363 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:31.598366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:31.598622 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:31.598881 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:31.598886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.599831 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.599832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:31.600660 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:31.600662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.601093 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.601095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:31.601235 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.601771 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:31.601780 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:31.603495 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:31.605077 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:31.605084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:31.605086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:31.605312 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:31.605535 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:31.605536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:31.605536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:31.605671 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:31.605928 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:31.605935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:31.606086 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:15:31.606197 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:15:31.606352 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
I20260925 15:15:31.607378 14126 heartbeater.cc:347] Connected to a master server at 127.8.43.190:46209
I20260925 15:15:31.607484 14126 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:31.607667 14126 heartbeater.cc:511] Master 127.8.43.190:46209 requested a full tablet report, sending...
I20260925 15:15:31.608309 13935 ts_manager.cc:194] Registered new tserver with Master: 7de6efcc5daf4e9dbf8117fb7c4b0bbc (127.8.43.129:46855)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
I20260925 15:15:31.609364 13935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:53429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:15:31.624325 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46209
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:31.704582 14131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:31.704794 14131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:31.704818 14131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:31.705979 14131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:31.706022 14131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:31.706048 14131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:31.707542 14131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:31.707796 14131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:31.708029 14131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:31.713061 14136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:31.713042 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:31.713107 14131 server_base.cc:1034] running on GCE node
W20260925 15:15:31.713016 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:31.713466 14131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:31.713717 14131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:31.714882 14131 hybrid_clock.cc:648] HybridClock initialized: now 1790349331714867 us; error 42 us; skew 500 ppm
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.717324 14131 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:15:31.717748 14131 webserver.cc:460] Webserver started at http://127.8.43.130:44587/ using document root <none> and password file <none>
I20260925 15:15:31.717998 14131 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:31.718070 14131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:31.718211 14131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:31.719343 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "054637d073f7480ea636066e85cf505c"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "d93224065433bff458dcd63b85da6978"
server_key_iv: "0ced091a97209061b6bbda42430f3daa"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.719702 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "054637d073f7480ea636066e85cf505c"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "d93224065433bff458dcd63b85da6978"
server_key_iv: "0ced091a97209061b6bbda42430f3daa"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.720992 14131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:15:31.721740 14146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.721915 14131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:31.722033 14131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "054637d073f7480ea636066e85cf505c"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "d93224065433bff458dcd63b85da6978"
server_key_iv: "0ced091a97209061b6bbda42430f3daa"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.722119 14131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:31.731419 14131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:31.733144 14131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:31.733301 14131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:31.733525 14131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:31.733810 14131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:31.733861 14131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.733911 14131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:31.733938 14131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.739814 14131 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:44059
I20260925 15:15:31.739883 14259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:44059 every 8 connection(s)
I20260925 15:15:31.740198 14131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:31.748833 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.740918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:46851 (local address 127.8.43.190:46209)
0925 15:15:31.741008 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.741011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.741180 (+ 169us) server_negotiation.cc:407] Connection header received
0925 15:15:31.742066 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.742067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.742077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.742131 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.742689 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.743169 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.744165 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.744345 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.744398 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.746445 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:31.746446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:31.746447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:31.746452 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:31.747660 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.748022 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.748023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.748023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.748081 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.748289 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.748289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.748290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.748384 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:31.748433 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:31.748656 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:15:31.748714 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:15:31.748929 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.740886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46209 (local address 127.8.43.130:46851)
0925 15:15:31.741052 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:31.741085 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:31.741972 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:31.742183 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:31.742186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:31.742431 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:31.742619 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:31.742623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.743294 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.743295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:31.744092 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:31.744094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.744415 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.744415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:31.744518 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.745045 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:31.745053 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:31.746366 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:31.747726 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:31.747731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:31.747733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:31.747939 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:31.748138 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:31.748138 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:31.748138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:31.748233 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:31.748437 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:31.748441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:31.748549 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 15:15:31.748630 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:15:31.748743 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260925 15:15:31.748940 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 14131
I20260925 15:15:31.749023 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:31.749193 8366 external_mini_cluster.cc:1428] Setting key f3180e2c7e1995de72f6fc11aff04352
I20260925 15:15:31.749540 14260 heartbeater.cc:347] Connected to a master server at 127.8.43.190:46209
I20260925 15:15:31.749677 14260 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:31.749881 14260 heartbeater.cc:511] Master 127.8.43.190:46209 requested a full tablet report, sending...
I20260925 15:15:31.750254 13935 ts_manager.cc:194] Registered new tserver with Master: 054637d073f7480ea636066e85cf505c (127.8.43.130:44059)
I20260925 15:15:31.750639 13935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:46851
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:15:31.774340 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46209
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:31.853965 14265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:31.854156 14265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:31.854220 14265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:31.855645 14265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:31.855707 14265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:31.855751 14265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:31.857307 14265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:31.857589 14265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:31.857810 14265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:31.862748 14271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:31.862788 14273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:31.862788 14270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:31.863034 14265 server_base.cc:1034] running on GCE node
I20260925 15:15:31.863344 14265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:31.863575 14265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:31.864733 14265 hybrid_clock.cc:648] HybridClock initialized: now 1790349331864703 us; error 45 us; skew 500 ppm
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.867038 14265 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:15:31.867511 14265 webserver.cc:460] Webserver started at http://127.8.43.131:36973/ using document root <none> and password file <none>
I20260925 15:15:31.867736 14265 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:31.867799 14265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:31.867929 14265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:31.868935 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "f8b34f4a45ba4692962aebe52fc42fde"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "8b7fc230da2282196978316e28a891a8"
server_key_iv: "e7fd8ad3fde80b4393500bc38d9dcae0"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.869293 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "f8b34f4a45ba4692962aebe52fc42fde"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "8b7fc230da2282196978316e28a891a8"
server_key_iv: "e7fd8ad3fde80b4393500bc38d9dcae0"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.870553 14265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:31.871300 14280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.871474 14265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:31.871569 14265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "f8b34f4a45ba4692962aebe52fc42fde"
format_stamp: "Formatted at 2026-09-25 15:15:31 on dist-test-slave-c5d6"
server_key: "8b7fc230da2282196978316e28a891a8"
server_key_iv: "e7fd8ad3fde80b4393500bc38d9dcae0"
server_key_version: "encryptionkey@0"
I20260925 15:15:31.871655 14265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:31.884554 14265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:31.886345 14265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:31.886510 14265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:31.886736 14265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:31.887053 14265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:31.887107 14265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.887148 14265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:31.887176 14265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:31.893128 14265 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:42661
I20260925 15:15:31.893173 14393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:42661 every 8 connection(s)
I20260925 15:15:31.893467 14265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:31.899024 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 14265
I20260925 15:15:31.899101 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:31.899351 8366 external_mini_cluster.cc:1428] Setting key a155e81af008a83343521b440282bb82
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:15:31.903398 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.894181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:46509 (local address 127.8.43.190:46209)
0925 15:15:31.894244 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.894246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.894458 (+ 212us) server_negotiation.cc:407] Connection header received
0925 15:15:31.895566 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.895567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.895580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.895630 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.896258 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.896889 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.898052 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.898236 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.898297 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.900909 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:31.900910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:31.900910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:31.900917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:31.902164 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.902551 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.902552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.902552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.902601 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.902824 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.902824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.902824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.902923 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:31.902971 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:31.903259 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:15:31.903334 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:15:31.903659 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.894173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46209 (local address 127.8.43.131:46509)
0925 15:15:31.894339 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:31.894374 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:31.895438 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:31.895696 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:31.895700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:31.895952 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:31.896153 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:31.896157 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.897007 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.897008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:31.897965 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:31.897967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.898319 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.898320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:31.898432 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.899094 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:31.899106 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:31.900818 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:31.902240 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:31.902246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:31.902248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:31.902476 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:31.902662 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:31.902663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:31.902663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:31.902765 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:31.902980 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:31.902984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:31.903107 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 15:15:31.903195 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:15:31.903387 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260925 15:15:31.904122 14394 heartbeater.cc:347] Connected to a master server at 127.8.43.190:46209
I20260925 15:15:31.904210 14394 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:31.904392 14394 heartbeater.cc:511] Master 127.8.43.190:46209 requested a full tablet report, sending...
I20260925 15:15:31.904764 13935 ts_manager.cc:194] Registered new tserver with Master: f8b34f4a45ba4692962aebe52fc42fde (127.8.43.131:42661)
I20260925 15:15:31.905148 13935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:46509
I20260925 15:15:31.911044 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/test-admin.keytab.
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.928171 8366 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 15:15:31.930042 14401 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:31.929194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.8.43.129:46855)
0925 15:15:31.929367 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.929373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.929386 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:15:31.929412 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.929414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.929428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.929503 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.929930 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55986: BlockingRecv error: recv got EOF from 127.0.0.1:55986 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
W20260925 15:15:31.930087 8366 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.931123 8366 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.932044 8366 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:31.939661 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.932564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.8.43.129:46855)
0925 15:15:31.932601 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.932603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.932656 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:15:31.932709 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.932710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.932720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.932768 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.933130 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.933903 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.934604 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.934931 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.935003 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.937008 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:31.937015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:31.937016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:31.937022 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:31.938552 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.938846 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.938848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.938848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.938903 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.939121 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.939121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.939121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.939312 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:31.939415 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:31.939535 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:15:31.939590 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 15:15:31.941591 14401 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:31.940917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.8.43.129:46855)
0925 15:15:31.940969 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.940970 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.941002 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:15:31.941075 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.941076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.941084 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.941126 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.941533 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55994: BlockingRecv error: recv got EOF from 127.0.0.1:55994 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 15:15:31.941606 8366 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.942534 8366 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.943397 8366 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:31.949834 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.943837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.8.43.129:46855)
0925 15:15:31.943920 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.943921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.943957 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:15:31.944020 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.944020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.944028 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.944072 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.944391 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.944865 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.945487 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.945659 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.945707 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.947573 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:31.947573 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:31.947574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:31.947579 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:31.948781 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.949126 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.949127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.949128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.949181 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.949354 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.949354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.949355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.949439 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:31.949506 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:31.949635 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 15:15:31.949692 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 15:15:31.951367 14401 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:15:31.950790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.8.43.129:46855)
0925 15:15:31.950865 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.950867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.950927 (+ 60us) server_negotiation.cc:407] Connection header received
0925 15:15:31.951003 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.951004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.951013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.951059 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.951317 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56006: BlockingRecv error: recv got EOF from 127.0.0.1:56006 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 15:15:31.951536 8366 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349306597827-8366-0/krb5kdc/krb5cc)
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.952464 8366 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:31.953258 8366 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:15:31 dist-test-slave-c5d6 krb5kdc[13890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:15:31.959954 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:31.953739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.8.43.129:46855)
0925 15:15:31.953796 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:31.953798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:31.953849 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:15:31.953919 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:31.953920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:31.953928 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:31.953996 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:31.954338 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.954840 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.955596 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:31.955770 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:31.955851 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:31.957758 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:31.957759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:31.957759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:31.957765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:31.959122 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.959446 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.959447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.959447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.959496 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:31.959639 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:31.959639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:31.959640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:31.959716 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:31.959776 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:31.959854 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:15:31.959902 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:15:31.960561 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13997
I20260925 15:15:31.965363 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 14131
I20260925 15:15:31.969758 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 14265
I20260925 15:15:31.973723 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 13903
2026-09-25T15:15:31Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1866 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:32 dist-test-slave-c5d6 krb5kdc[14414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:32 dist-test-slave-c5d6 krb5kdc[14414](info): set up 2 sockets
Sep 25 15:15:32 dist-test-slave-c5d6 krb5kdc[14414](info): commencing operation
krb5kdc: starting...
W20260925 15:15:33.015589 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.190.keytab.
Entry for principal oryx/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.190.keytab.
I20260925 15:15:33.127032 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:34445
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.190.keytab
--principal=oryx/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:34445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:33.205135 14427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:33.205377 14427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:33.205412 14427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:33.206540 14427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:33.206609 14427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:33.206633 14427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:33.206661 14427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:33.206692 14427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:33.208122 14427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:34445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.190.keytab
--principal=oryx/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:34445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:33.208379 14427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:33.208611 14427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:33.214035 14435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.214035 14432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.214105 14433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:33.214289 14427 server_base.cc:1034] running on GCE node
I20260925 15:15:33.214551 14427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:33.214836 14427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:33.216001 14427 hybrid_clock.cc:648] HybridClock initialized: now 1790349333215992 us; error 45 us; skew 500 ppm
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:33.218442 14427 init.cc:377] Logged in from keytab as oryx/127.8.43.190@KRBTEST.COM (short username oryx)
I20260925 15:15:33.218904 14427 webserver.cc:460] Webserver started at http://127.8.43.190:35287/ using document root <none> and password file <none>
I20260925 15:15:33.219147 14427 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:33.219218 14427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:33.219391 14427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:33.220438 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "743e6dbca0ce4f3da82a3d4d3371b902"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "87138514b5e68131654d39ba6965ed44"
server_key_iv: "b312324beebeb31b27de0e8e3ab0e6ec"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.220788 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "743e6dbca0ce4f3da82a3d4d3371b902"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "87138514b5e68131654d39ba6965ed44"
server_key_iv: "b312324beebeb31b27de0e8e3ab0e6ec"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.222146 14427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:33.222810 14442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.222975 14427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:33.223073 14427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "743e6dbca0ce4f3da82a3d4d3371b902"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "87138514b5e68131654d39ba6965ed44"
server_key_iv: "b312324beebeb31b27de0e8e3ab0e6ec"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.223156 14427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:33.236267 14427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:33.237996 14427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:33.238183 14427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:33.242236 14427 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:34445
I20260925 15:15:33.242280 14494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:34445 every 8 connection(s)
I20260925 15:15:33.242642 14427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:33.243700 14495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:33.246366 14495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: Bootstrap starting.
I20260925 15:15:33.246928 14495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:33.247328 14495 log.cc:824] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:33.248473 14495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: No bootstrap required, opened a new log
I20260925 15:15:33.250005 14495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } }
I20260925 15:15:33.250138 14495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:33.250160 14495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743e6dbca0ce4f3da82a3d4d3371b902, State: Initialized, Role: FOLLOWER
I20260925 15:15:33.250275 14495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } }
I20260925 15:15:33.250325 14495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:33.250345 14495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:33.250372 14495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:33.251096 14495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } }
I20260925 15:15:33.251191 14495 leader_election.cc:302] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 743e6dbca0ce4f3da82a3d4d3371b902; no voters:
I20260925 15:15:33.251370 14495 leader_election.cc:288] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:33.251459 14498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:33.251600 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 14427
I20260925 15:15:33.251638 14495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:33.251685 14498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 1 LEADER]: Becoming Leader. State: Replica: 743e6dbca0ce4f3da82a3d4d3371b902, State: Running, Role: LEADER
I20260925 15:15:33.251724 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:15:33.251801 14498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } }
I20260925 15:15:33.251930 8366 external_mini_cluster.cc:1428] Setting key ad39af3e9fccab1b4f671390434fc76e
I20260925 15:15:33.252125 14498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743e6dbca0ce4f3da82a3d4d3371b902. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } }
I20260925 15:15:33.252192 14498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:33.252156 14499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } }
I20260925 15:15:33.252240 14499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: This master's current role is: LEADER
W20260925 15:15:33.253340 14515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:15:33.253428 14515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:15:33.253515 14507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:33.253684 14507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:33.255136 14507 catalog_manager.cc:1347] Generated new cluster ID: 44c0c811054c47b9a2e16757b2d002f6
I20260925 15:15:33.255182 14507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:15:33.264130 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.252589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.8.43.190:34445)
0925 15:15:33.252745 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.252754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.252784 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:15:33.252900 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.252907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.252965 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.253902 (+ 937us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.253972 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.255305 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.256350 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.256938 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.257031 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.259806 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.259810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.259811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.259822 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.262457 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.262836 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.262839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.262839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.262976 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.263176 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.263177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.263178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.263527 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.263689 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.263765 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.263903 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260925 15:15:33.268357 14507 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:33.268843 14507 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:33.277086 14507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: Generated new TSK 0
I20260925 15:15:33.277315 14507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab.
Entry for principal oryx/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab.
I20260925 15:15:33.302129 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34445
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab
--principal=oryx/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:33.382329 14521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:33.382570 14521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:33.382607 14521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:33.383783 14521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:33.383847 14521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:33.383893 14521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:33.385334 14521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab
--principal=oryx/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:33.385604 14521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:33.385830 14521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:33.390726 14529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.390731 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.390832 14526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:33.390983 14521 server_base.cc:1034] running on GCE node
I20260925 15:15:33.391271 14521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:33.391472 14521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:33.392678 14521 hybrid_clock.cc:648] HybridClock initialized: now 1790349333392662 us; error 29 us; skew 500 ppm
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:33.395200 14521 init.cc:377] Logged in from keytab as oryx/127.8.43.129@KRBTEST.COM (short username oryx)
I20260925 15:15:33.395773 14521 webserver.cc:460] Webserver started at http://127.8.43.129:34033/ using document root <none> and password file <none>
I20260925 15:15:33.396032 14521 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:33.396112 14521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:33.396270 14521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:33.397502 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "0c0cc0eaf40745c3b1301f97cf484511"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "b85383f2ca4da20eb471ef32e3c43cde"
server_key_iv: "106ce0af9468111fbb2dfc641d964f00"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.397895 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "0c0cc0eaf40745c3b1301f97cf484511"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "b85383f2ca4da20eb471ef32e3c43cde"
server_key_iv: "106ce0af9468111fbb2dfc641d964f00"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.399333 14521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:33.400090 14536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.400262 14521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:33.400372 14521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "0c0cc0eaf40745c3b1301f97cf484511"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "b85383f2ca4da20eb471ef32e3c43cde"
server_key_iv: "106ce0af9468111fbb2dfc641d964f00"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.400456 14521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:33.413196 14521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:33.415759 14521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:33.415940 14521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:33.416186 14521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:33.416523 14521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:33.416582 14521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.416627 14521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:33.416653 14521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.422788 14521 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:38371
I20260925 15:15:33.422827 14649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:38371 every 8 connection(s)
I20260925 15:15:33.423138 14521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:33.426955 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 14521
I20260925 15:15:33.427073 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:33.427290 8366 external_mini_cluster.cc:1428] Setting key 9279a9d8e06788249e5bc518c9ee16f4
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:15:33.432775 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.423835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:35049 (local address 127.8.43.190:34445)
0925 15:15:33.423926 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.423929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.424133 (+ 204us) server_negotiation.cc:407] Connection header received
0925 15:15:33.424604 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.424605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.424615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.424675 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.425572 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.426123 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.427457 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.427726 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.427818 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.429948 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.429950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.429950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.429958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.431394 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.431820 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.431821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.431822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.431883 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.432140 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.432141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.432142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.432260 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.432337 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.432604 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.432686 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:15:33.432967 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.423813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:35049)
0925 15:15:33.424005 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.424040 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.424511 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.424731 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.424734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.425299 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.425491 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.425495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.426228 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.426229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.427373 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.427376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.427827 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.427828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.427964 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.428514 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.428522 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.429852 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.431487 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.431495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.431497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.431730 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.431950 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.431951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.431952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.432070 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.432328 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.432335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.432473 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.432583 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.432735 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260925 15:15:33.433446 14650 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34445
I20260925 15:15:33.433554 14650 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:33.433723 14650 heartbeater.cc:511] Master 127.8.43.190:34445 requested a full tablet report, sending...
I20260925 15:15:33.434269 14459 ts_manager.cc:194] Registered new tserver with Master: 0c0cc0eaf40745c3b1301f97cf484511 (127.8.43.129:38371)
I20260925 15:15:33.435242 14459 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:35049
WARNING: no policy specified for oryx/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab.
Entry for principal oryx/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab.
I20260925 15:15:33.451807 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34445
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab
--principal=oryx/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:33.540632 14655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:33.540879 14655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:33.540904 14655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:33.542189 14655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:33.542236 14655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:33.542266 14655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:33.543911 14655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab
--principal=oryx/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:33.544169 14655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:33.544430 14655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:33.549853 14663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.549880 14660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.549893 14661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:33.550065 14655 server_base.cc:1034] running on GCE node
I20260925 15:15:33.550278 14655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:33.550599 14655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:33.551802 14655 hybrid_clock.cc:648] HybridClock initialized: now 1790349333551781 us; error 37 us; skew 500 ppm
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:33.554536 14655 init.cc:377] Logged in from keytab as oryx/127.8.43.130@KRBTEST.COM (short username oryx)
I20260925 15:15:33.555002 14655 webserver.cc:460] Webserver started at http://127.8.43.130:34619/ using document root <none> and password file <none>
I20260925 15:15:33.555303 14655 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:33.555374 14655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:33.555513 14655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:33.556566 14655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "aaaf0e4f20874f49b5a970a66e72f0b0"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "196a40a723132381e1c95bb4a01b5892"
server_key_iv: "9b6d046e1374db1449810ee34aa76435"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.556924 14655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "aaaf0e4f20874f49b5a970a66e72f0b0"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "196a40a723132381e1c95bb4a01b5892"
server_key_iv: "9b6d046e1374db1449810ee34aa76435"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.558228 14655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:15:33.558986 14670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.559175 14655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:33.559294 14655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "aaaf0e4f20874f49b5a970a66e72f0b0"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "196a40a723132381e1c95bb4a01b5892"
server_key_iv: "9b6d046e1374db1449810ee34aa76435"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.559378 14655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:33.576555 14655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:33.578902 14655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:33.579071 14655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:33.579324 14655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:33.579639 14655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:33.579691 14655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.579726 14655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:33.579772 14655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.585637 14655 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:39207
I20260925 15:15:33.585678 14783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:39207 every 8 connection(s)
I20260925 15:15:33.586023 14655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:33.586690 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 14655
I20260925 15:15:33.586798 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:33.587041 8366 external_mini_cluster.cc:1428] Setting key 33406a8d093909abcbe3719e8a3172b8
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.43.131@KRBTEST.COM; defaulting to no policy
I20260925 15:15:33.596436 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.586866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:54721 (local address 127.8.43.190:34445)
0925 15:15:33.586937 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.586941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.587103 (+ 162us) server_negotiation.cc:407] Connection header received
0925 15:15:33.587776 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.587777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.587791 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.587859 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.588834 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.589522 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.590804 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.591155 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.591246 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.593427 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.593429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.593429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.593437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.595039 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.595492 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.595494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.595495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.595559 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.595825 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.595826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.595826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.595949 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.596009 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.596302 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.596371 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:15:33.596565 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.586788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:54721)
0925 15:15:33.586981 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.587022 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.587652 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.587945 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.587950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.588536 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.588722 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.588727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.589614 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.589615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.590697 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.590699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.591296 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.591297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.591442 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.591957 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.591973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.593330 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.595135 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.595140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.595142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.595414 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.595636 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.595636 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.595637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.595745 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.596007 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.596028 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.596142 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.596221 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.596337 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260925 15:15:33.597205 14784 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34445
I20260925 15:15:33.597302 14784 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:33.597488 14784 heartbeater.cc:511] Master 127.8.43.190:34445 requested a full tablet report, sending...
I20260925 15:15:33.597908 14459 ts_manager.cc:194] Registered new tserver with Master: aaaf0e4f20874f49b5a970a66e72f0b0 (127.8.43.130:39207)
I20260925 15:15:33.598287 14459 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:54721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab.
Entry for principal oryx/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab.
I20260925 15:15:33.614296 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34445
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab
--principal=oryx/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:33.694550 14789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:33.694782 14789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:33.694818 14789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:33.695989 14789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:33.696062 14789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:33.696105 14789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:33.697538 14789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab
--principal=oryx/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:34445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:33.697804 14789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:33.698035 14789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:15:33.703012 14789 server_base.cc:1034] running on GCE node
W20260925 15:15:33.703015 14795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.702994 14794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.702994 14797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:33.703481 14789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:33.703786 14789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:33.704947 14789 hybrid_clock.cc:648] HybridClock initialized: now 1790349333704916 us; error 45 us; skew 500 ppm
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:33.707556 14789 init.cc:377] Logged in from keytab as oryx/127.8.43.131@KRBTEST.COM (short username oryx)
I20260925 15:15:33.707978 14789 webserver.cc:460] Webserver started at http://127.8.43.131:43017/ using document root <none> and password file <none>
I20260925 15:15:33.708199 14789 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:33.708264 14789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:33.708413 14789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:33.709523 14789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "c1c7eaab79f14061bdf0381a5778406e"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "cce693c07aa3f5235235a2adcb242dd4"
server_key_iv: "1b6f4144b5cdd6c951d4ff3028835949"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.709883 14789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "c1c7eaab79f14061bdf0381a5778406e"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "cce693c07aa3f5235235a2adcb242dd4"
server_key_iv: "1b6f4144b5cdd6c951d4ff3028835949"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.711177 14789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:15:33.711931 14804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.712090 14789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:33.712196 14789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "c1c7eaab79f14061bdf0381a5778406e"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "cce693c07aa3f5235235a2adcb242dd4"
server_key_iv: "1b6f4144b5cdd6c951d4ff3028835949"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.712275 14789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:33.741001 14789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:33.743408 14789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:33.743635 14789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:33.743894 14789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:33.744206 14789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:33.744238 14789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.744282 14789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:33.744309 14789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.750201 14789 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:42813
I20260925 15:15:33.750267 14917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:42813 every 8 connection(s)
I20260925 15:15:33.750603 14789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:15:33.759454 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 14789
I20260925 15:15:33.759557 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:33.759773 8366 external_mini_cluster.cc:1428] Setting key e6ccb9ea5089df09781f8887e10e07fe
I20260925 15:15:33.760546 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.751377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:58193 (local address 127.8.43.190:34445)
0925 15:15:33.751448 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.751450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.751806 (+ 356us) server_negotiation.cc:407] Connection header received
0925 15:15:33.752489 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.752491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.752504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.752593 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.753736 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.754235 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.755255 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.755442 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.755498 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.757498 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.757499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.757499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.757506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.758863 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.759306 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.759307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.759308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.759376 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.759722 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.759724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.759724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.759869 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.759948 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.760359 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.760441 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:15:33.760777 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.751407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:58193)
0925 15:15:33.751598 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.751639 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.752246 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.752675 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.752679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.753447 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.753640 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.753645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.754323 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.754324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.755144 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.755145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.755518 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.755518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.755618 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.756046 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.756054 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.757386 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.758943 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.758949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.758950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.759171 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.759459 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.759460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.759460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.759662 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.760015 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.760020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.760197 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.760312 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.760478 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260925 15:15:33.761225 14918 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34445
I20260925 15:15:33.761324 14918 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:33.761534 14918 heartbeater.cc:511] Master 127.8.43.190:34445 requested a full tablet report, sending...
I20260925 15:15:33.761982 14459 ts_manager.cc:194] Registered new tserver with Master: c1c7eaab79f14061bdf0381a5778406e (127.8.43.131:42813)
I20260925 15:15:33.762311 14459 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:58193
I20260925 15:15:33.771633 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.43.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.189.keytab.
Entry for principal oryx/127.8.43.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.189.keytab.
I20260925 15:15:33.796140 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.189:42655
--webserver_interface=127.8.43.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.189.keytab
--principal=oryx/127.8.43.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:34445,127.8.43.189:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:33.875171 14923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:33.875505 14923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:33.875540 14923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:33.877667 14923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:33.877727 14923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:33.877751 14923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:33.877780 14923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:33.877801 14923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:33.880204 14923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:34445,127.8.43.189:42655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.189.keytab
--principal=oryx/127.8.43.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.189:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.43.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
W20260925 15:15:33.880369 14923 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.43.190:34445,127.8.43.189:42655'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 15:15:33.880533 14923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:33.880800 14923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:33.885885 14929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.886132 14928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:33.891366 14931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:33.893618 14923 server_base.cc:1034] running on GCE node
I20260925 15:15:33.893867 14923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:33.894138 14923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:33.895290 14923 hybrid_clock.cc:648] HybridClock initialized: now 1790349333895277 us; error 41 us; skew 500 ppm
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:33.897897 14923 init.cc:377] Logged in from keytab as oryx/127.8.43.189@KRBTEST.COM (short username oryx)
I20260925 15:15:33.898298 14923 webserver.cc:460] Webserver started at http://127.8.43.189:41641/ using document root <none> and password file <none>
I20260925 15:15:33.898483 14923 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:33.898523 14923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:33.898615 14923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:33.899786 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data/instance:
uuid: "863d08373b7a4f6bb257e32b07b66d89"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "85264a727a8d25e33b4ab3900510937b"
server_key_iv: "378c9b21a10ba79ee89a30dcaec7b39e"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.900096 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal/instance:
uuid: "863d08373b7a4f6bb257e32b07b66d89"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "85264a727a8d25e33b4ab3900510937b"
server_key_iv: "378c9b21a10ba79ee89a30dcaec7b39e"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.901432 14923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:15:33.902146 14938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.902364 14923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:33.902458 14923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
uuid: "863d08373b7a4f6bb257e32b07b66d89"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "85264a727a8d25e33b4ab3900510937b"
server_key_iv: "378c9b21a10ba79ee89a30dcaec7b39e"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.902532 14923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:33.917084 14923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:33.919435 14923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:33.919580 14923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:33.923511 14923 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.189:42655
I20260925 15:15:33.923629 14990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.189:42655 every 8 connection(s)
I20260925 15:15:33.923934 14923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
I20260925 15:15:33.924767 14991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:33.926309 14991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:15:33.930932 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 14923
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.189@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:15:33.935865 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.927049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.8.43.190:34445)
0925 15:15:33.927167 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.927170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.927413 (+ 243us) server_negotiation.cc:407] Connection header received
0925 15:15:33.927912 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.927914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.927925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.927976 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.928632 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.929124 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.930154 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.930339 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.930426 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.932992 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.932994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.932994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.933000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.934346 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.934778 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.934779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.934780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.934847 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.935120 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.935121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.935122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.935327 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.935411 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.935696 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.935775 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:15:33.936066 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.927015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33116)
0925 15:15:33.927288 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.927330 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.927808 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.928071 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.928075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.928309 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.928493 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.928496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.929216 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.929217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.930026 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.930028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.930450 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.930451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.930570 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.931130 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.931145 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.932910 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.934416 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.934420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.934422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.934701 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.934915 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.934915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.934916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.935044 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.935400 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.935406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.935540 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.935676 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.935823 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.43.189@KRBTEST.COM
I20260925 15:15:33.936762 14991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:15:33.939927 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.931453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.8.43.189:42655)
0925 15:15:33.931575 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.931579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.931686 (+ 107us) server_negotiation.cc:407] Connection header received
0925 15:15:33.931748 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.931750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.931775 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.931838 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.932567 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.933442 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.934150 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.934486 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.934554 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.936969 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.936975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.936976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.936984 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.938520 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.938892 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.938895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.938896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.938951 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.939155 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.939156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.939156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.939350 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.939518 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.939571 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.939640 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.189@KRBTEST.COM for oryx/127.8.43.189@KRBTEST.COM
I20260925 15:15:33.943109 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.937019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.0.0.1:35850)
0925 15:15:33.937075 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.937085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.937123 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.937330 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.937331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.937469 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.937508 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.937509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.938136 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.938137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.938597 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.938598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.938975 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.938982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.939043 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.939439 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.939443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.940647 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.942335 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.942335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.942336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.942495 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.942698 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.942698 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.942698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.942742 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.942895 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.942896 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.942925 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.942968 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.943046 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:15:33.943225 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.937063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.8.43.189:42655)
0925 15:15:33.937209 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.937210 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.937218 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:33.937236 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.937236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.937251 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.937329 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.937579 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.938047 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.938686 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.938893 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.938974 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.940716 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.940717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.940717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.940723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.942262 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.942572 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.942573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.942573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.942633 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.942779 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.942779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.942779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.942860 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.942904 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.942984 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.943107 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260925 15:15:33.944388 14991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Bootstrap starting.
I20260925 15:15:33.945000 14991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:33.945267 14991 log.cc:824] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:33.945875 14991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: No bootstrap required, opened a new log
I20260925 15:15:33.947178 14991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } }
I20260925 15:15:33.947361 14991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:33.947427 14991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863d08373b7a4f6bb257e32b07b66d89, State: Initialized, Role: FOLLOWER
I20260925 15:15:33.947554 14991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } }
I20260925 15:15:33.947855 14991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:33.947846 14998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } } }
I20260925 15:15:33.947947 14998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:15:33.949544 15009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:15:33.949601 15009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:15:33.951503 14923 master_runner.cc:428] Detected that this master 863d08373b7a4f6bb257e32b07b66d89 is joining an existing cluster
I20260925 15:15:33.951572 14923 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.43.189:42655
I20260925 15:15:33.959951 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.952719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.0.0.1:35866)
0925 15:15:33.952864 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.952874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.952936 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.953107 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.953108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.953320 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.953412 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.953414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.954296 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.954297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.954900 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.954902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.955688 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.955689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.955737 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.956322 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.956326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.956662 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.958733 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.958735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.958736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.959023 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.959256 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.959257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.959257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.959322 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.959591 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.959592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.959628 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.959721 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.959830 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260925 15:15:33.960112 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.953020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.8.43.190:34445)
0925 15:15:33.953079 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.953081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.953325 (+ 244us) server_negotiation.cc:407] Connection header received
0925 15:15:33.953425 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.953426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.953451 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.953503 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.953810 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.954299 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.955371 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.955585 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.955642 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.956857 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.956858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.956859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.956866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.958716 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.959104 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.959106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.959106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.959161 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.959453 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.959454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.959454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.959601 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.959674 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.959831 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.959906 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:15:33.959976 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.953094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33128)
0925 15:15:33.953252 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.953262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.953353 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.953545 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.953546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.953678 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.953721 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.953723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.954381 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.954382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.955284 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.955286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.955678 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.955679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.955743 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.956235 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.956240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.956710 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.958780 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.958781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.958781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.959017 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.959261 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.959261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.959262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.959341 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.959673 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.959674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.959710 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.959800 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.959876 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260925 15:15:33.960512 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.952749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.8.43.189:42655)
0925 15:15:33.952791 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.952793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.952942 (+ 149us) server_negotiation.cc:407] Connection header received
0925 15:15:33.952993 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.952994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.953005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.953065 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.953507 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.954210 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.955011 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.955297 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.955381 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.956766 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.956768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.956768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.956775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.958632 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.959107 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.959108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.959108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.959157 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.959389 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.959390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.959390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.959524 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.959599 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.959755 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.960430 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":598,"server-negotiator.queue_time_us":21}
Sep 25 15:15:33 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.190@KRBTEST.COM for oryx/127.8.43.189@KRBTEST.COM
I20260925 15:15:33.969568 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.962344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.0.0.1:35872)
0925 15:15:33.962493 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.962532 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.962626 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.962749 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.962752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.963051 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.963095 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.963096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.964090 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.964090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.964623 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.964624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.965063 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.965063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.965125 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.965604 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.965608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.966993 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.968644 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.968650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.968651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.968829 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.969024 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.969024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.969025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.969073 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.969312 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.969317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.969355 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.969416 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.969492 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260925 15:15:33.969635 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.962413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.8.43.189:42655)
0925 15:15:33.962488 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.962490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.962602 (+ 112us) server_negotiation.cc:407] Connection header received
0925 15:15:33.962689 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.962690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.962699 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.962746 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.963276 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.964005 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.964696 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.964975 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.965061 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.967080 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.967082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.967082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.967088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.968556 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.968899 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.968900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.968900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.968956 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.969132 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.969132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.969133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.969250 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.969316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.969475 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.969551 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:15:33.970233 14459 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.43.189:42655: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89"
member_type: NON_VOTER
last_known_addr {
host: "127.8.43.189"
port: 42655
}
attrs {
promote: true
}
}
dest_uuid: "743e6dbca0ce4f3da82a3d4d3371b902"
cas_config_opid_index: -1
I20260925 15:15:33.970536 14459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } }
I20260925 15:15:33.971064 15019 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 863d08373b7a4f6bb257e32b07b66d89 (127.8.43.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } }
I20260925 15:15:33.971410 14965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:33.972064 15019 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.43.189:42655
I20260925 15:15:33.972074 14965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Refusing update from remote peer 743e6dbca0ce4f3da82a3d4d3371b902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 15:15:33.972136 15019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } }
I20260925 15:15:33.972188 15019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:33.972329 14923 master.cc:537] Master@127.8.43.189:42655 shutting down...
I20260925 15:15:33.972357 15019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:15:33.973011 14443 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 15:15:33.973454 14443 consensus_peers.cc:487] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 -> Peer 863d08373b7a4f6bb257e32b07b66d89 (127.8.43.189:42655): Couldn't send request to peer 863d08373b7a4f6bb257e32b07b66d89. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:15:33.974798 14923 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:33.974881 14923 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:33.974931 14923 tablet_replica.cc:330] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: stopping tablet replica
I20260925 15:15:33.988010 14923 master.cc:552] Master@127.8.43.189:42655 shutdown complete.
I20260925 15:15:33.988067 14923 master_runner.cc:305] Clearing existing system tablet
I20260925 15:15:33.988200 14923 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:33.988250 14923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:33.988965 14923 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260925 15:15:33.989523 15023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:33.989642 14923 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:33.989699 14923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:33.989722 14923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
uuid: "863d08373b7a4f6bb257e32b07b66d89"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "85264a727a8d25e33b4ab3900510937b"
server_key_iv: "378c9b21a10ba79ee89a30dcaec7b39e"
server_key_version: "encryptionkey@0"
I20260925 15:15:33.990432 14923 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:33.991957 14923 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:15:33.992041 14923 log.cc:1196] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 15:15:33.992247 14923 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Deleting consensus metadata
I20260925 15:15:33.992399 14923 master_runner.cc:315] Copying system tablet from 127.8.43.190:34445
I20260925 15:15:33.993036 14923 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: tablet copy: Beginning tablet copy session from remote peer at address 127.8.43.190:34445
I20260925 15:15:33.999642 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.993345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33142)
0925 15:15:33.993513 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:33.993522 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:33.993576 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:33.993719 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:33.993719 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:33.993862 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:33.993914 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.993916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.994865 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.994866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.995511 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:33.995513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.995994 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.995995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:33.996072 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.996609 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:33.996614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:33.996924 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:33.998626 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.998627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.998627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.998793 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.998991 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:33.998991 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:33.998991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:33.999051 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:33.999374 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:33.999375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:33.999413 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:15:33.999482 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:15:33.999568 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260925 15:15:33.999703 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:33.993394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.8.43.190:34445)
0925 15:15:33.993462 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:33.993464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:33.993583 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:15:33.993635 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:33.993643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:33.993655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:33.993717 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:33.993977 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.994785 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.995589 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:33.995897 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:33.995970 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:33.997015 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:33.997016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:33.997017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:33.997025 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:33.998545 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.998866 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.998868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.998868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.998928 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:33.999114 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:33.999115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:33.999115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:33.999305 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:33.999377 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:33.999536 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:15:33.999614 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:15:34.000124 14479 tablet_copy_service.cc:140] P 743e6dbca0ce4f3da82a3d4d3371b902: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 863d08373b7a4f6bb257e32b07b66d89 ({username='oryx', principal='oryx/127.8.43.189@KRBTEST.COM'} at 127.0.0.1:33142)
I20260925 15:15:34.000284 14479 tablet_copy_service.cc:161] P 743e6dbca0ce4f3da82a3d4d3371b902: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 863d08373b7a4f6bb257e32b07b66d89 at {username='oryx', principal='oryx/127.8.43.189@KRBTEST.COM'} at 127.0.0.1:33142: session id = 863d08373b7a4f6bb257e32b07b66d89-00000000000000000000000000000000
I20260925 15:15:34.000936 14479 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:15:34.001590 14923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:15:34.002599 14923 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: tablet copy: Starting download of 0 data blocks...
I20260925 15:15:34.002784 14923 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: tablet copy: Starting download of 1 WAL segments...
I20260925 15:15:34.003749 14923 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:15:34.004570 14479 tablet_copy_service.cc:342] P 743e6dbca0ce4f3da82a3d4d3371b902: Request end of tablet copy session 863d08373b7a4f6bb257e32b07b66d89-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.43.189@KRBTEST.COM'} at 127.0.0.1:33142
I20260925 15:15:34.004637 14479 tablet_copy_service.cc:433] P 743e6dbca0ce4f3da82a3d4d3371b902: ending tablet copy session 863d08373b7a4f6bb257e32b07b66d89-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 863d08373b7a4f6bb257e32b07b66d89
I20260925 15:15:34.007169 14923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:34.008324 15031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.008352 15030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.008514 15033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:34.008672 14923 server_base.cc:1034] running on GCE node
I20260925 15:15:34.008759 14923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:34.008914 14923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:34.010046 14923 hybrid_clock.cc:648] HybridClock initialized: now 1790349334010036 us; error 24 us; skew 500 ppm
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:34.011494 14923 init.cc:377] Logged in from keytab as oryx/127.8.43.189@KRBTEST.COM (short username oryx)
I20260925 15:15:34.011802 14923 webserver.cc:460] Webserver started at http://127.8.43.189:36937/ using document root <none> and password file <none>
I20260925 15:15:34.011925 14923 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:34.011989 14923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:34.012594 14923 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:34.013129 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.013257 14923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:34.013303 14923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
uuid: "863d08373b7a4f6bb257e32b07b66d89"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "85264a727a8d25e33b4ab3900510937b"
server_key_iv: "378c9b21a10ba79ee89a30dcaec7b39e"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.013360 14923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:34.037070 14923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:34.038895 14923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:34.042351 14923 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.189:42655
I20260925 15:15:34.042369 15092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.189:42655 every 8 connection(s)
I20260925 15:15:34.042726 14923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
I20260925 15:15:34.043716 15093 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:15:34.044245 15093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Bootstrap starting.
I20260925 15:15:34.047457 15093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:15:34.047708 15093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: Bootstrap complete.
I20260925 15:15:34.047904 15093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } }
I20260925 15:15:34.047993 15093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 863d08373b7a4f6bb257e32b07b66d89, State: Initialized, Role: LEARNER
I20260925 15:15:34.048032 15093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } }
I20260925 15:15:34.048211 15093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:34.048203 15096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } }
I20260925 15:15:34.048317 15096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:15:34.049806 15107 catalog_manager.cc:1259] Loaded cluster ID: 44c0c811054c47b9a2e16757b2d002f6
I20260925 15:15:34.049875 15107 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: loading cluster ID for follower catalog manager: success
I20260925 15:15:34.051375 15107 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: acquiring CA information for follower catalog manager: success
I20260925 15:15:34.051703 15107 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:15:34.098155 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.092257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.8.43.189:42655)
0925 15:15:34.092398 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.092400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.092408 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:34.092431 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.092432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.092440 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.092492 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.093074 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.093738 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.094465 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.094662 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.094717 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.095903 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.095904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.095904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.095909 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.097157 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.097467 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.097468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.097468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.097522 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.097680 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.097680 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.097680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.097774 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.097844 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.097947 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.098021 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.8.43.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.188.keytab.
Entry for principal oryx/127.8.43.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.188.keytab.
I20260925 15:15:34.122733 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.188:39137
--webserver_interface=127.8.43.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.188.keytab
--principal=oryx/127.8.43.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:34.204726 15111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:34.204918 15111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:34.204941 15111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:34.206030 15111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:34.206069 15111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:34.206082 15111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:34.206094 15111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:34.206106 15111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:34.207574 15111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.188.keytab
--principal=oryx/127.8.43.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.188:39137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.43.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:34.207813 15111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:34.208031 15111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:34.212708 15116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.212661 15119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.212662 15117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:34.213007 15111 server_base.cc:1034] running on GCE node
I20260925 15:15:34.213208 15111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:34.213537 15111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:34.214686 15111 hybrid_clock.cc:648] HybridClock initialized: now 1790349334214674 us; error 27 us; skew 500 ppm
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:34.217272 15111 init.cc:377] Logged in from keytab as oryx/127.8.43.188@KRBTEST.COM (short username oryx)
I20260925 15:15:34.217654 15111 webserver.cc:460] Webserver started at http://127.8.43.188:40077/ using document root <none> and password file <none>
I20260925 15:15:34.217828 15111 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:34.217867 15111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:34.217953 15111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:34.218945 15111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data/instance:
uuid: "9720f9afeca241b6a20e6489603d4d25"
format_stamp: "Formatted at 2026-09-25 15:15:34 on dist-test-slave-c5d6"
server_key: "ff94dc3c065a163f7322b00e43f715a7"
server_key_iv: "524590279ce177c8a64a04d6fbfe7bf2"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.219319 15111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal/instance:
uuid: "9720f9afeca241b6a20e6489603d4d25"
format_stamp: "Formatted at 2026-09-25 15:15:34 on dist-test-slave-c5d6"
server_key: "ff94dc3c065a163f7322b00e43f715a7"
server_key_iv: "524590279ce177c8a64a04d6fbfe7bf2"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.220614 15111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:34.221346 15126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.221590 15111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:34.221685 15111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
uuid: "9720f9afeca241b6a20e6489603d4d25"
format_stamp: "Formatted at 2026-09-25 15:15:34 on dist-test-slave-c5d6"
server_key: "ff94dc3c065a163f7322b00e43f715a7"
server_key_iv: "524590279ce177c8a64a04d6fbfe7bf2"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.221760 15111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:34.238619 15111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:34.241006 15111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:34.241166 15111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:34.245497 15111 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.188:39137
I20260925 15:15:34.245522 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.188:39137 every 8 connection(s)
I20260925 15:15:34.245867 15111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
I20260925 15:15:34.246762 15179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:34.247387 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 15111
I20260925 15:15:34.248704 15179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.188@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.43.188@KRBTEST.COM
I20260925 15:15:34.258983 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.249712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.8.43.190:34445)
0925 15:15:34.249980 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.249982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.249990 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:34.250031 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.250032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.250042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.250094 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.250677 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.251386 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.252622 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.252845 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.252928 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.255480 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.255481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.255481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.255488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.257333 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.257862 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.257864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.257865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.257932 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.258223 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.258240 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.258240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.258393 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.258465 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.258782 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.258879 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260925 15:15:34.259158 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.249642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33150)
0925 15:15:34.249836 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.249874 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.249936 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.250115 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.250118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.250427 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.250535 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.250540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.251501 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.251502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.252503 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.252505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.252936 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.252938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.253046 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.253673 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.253683 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.255405 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.257418 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.257423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.257424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.257703 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.258028 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.258029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.258029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.258154 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.258457 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.258461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.258615 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.258730 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.258887 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":119,"threads_started":1}
I20260925 15:15:34.259300 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.247992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.8.43.188:39137)
0925 15:15:34.248154 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.248221 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.248254 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:15:34.248831 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.248840 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.248937 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.249106 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.249978 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.251354 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.252461 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.252973 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.253044 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.256109 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.256115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.256115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.256122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.257895 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.258268 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.258271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.258271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.258319 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.258532 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.258532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.258533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.258683 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.258836 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.258908 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.258970 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260925 15:15:34.259986 15179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.188@KRBTEST.COM for oryx/127.8.43.189@KRBTEST.COM
I20260925 15:15:34.265736 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.260217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.0.0.1:35914)
0925 15:15:34.260262 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.260272 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.260332 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.260461 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.260462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.260593 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.260632 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.260634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.261329 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.261330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.261860 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.261861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.262248 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.262249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.262313 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.262642 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.262646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.263754 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.264990 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.264991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.264991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.265141 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.265302 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.265302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.265303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.265346 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.265534 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.265535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.265566 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.265615 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.265677 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:15:34.265748 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.260290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.8.43.189:42655)
0925 15:15:34.260351 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.260353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.260361 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:34.260389 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.260390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.260402 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.260457 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.260717 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.261231 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.261954 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.262171 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.262249 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.263825 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.263826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.263826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.263832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.264925 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.265200 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.265201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.265201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.265250 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.265389 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.265389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.265389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.265471 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.265533 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.265632 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.265690 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:15:34.266142 15179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39137 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.188@KRBTEST.COM for oryx/127.8.43.188@KRBTEST.COM
I20260925 15:15:34.271905 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.266398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39137 (local address 127.0.0.1:54970)
0925 15:15:34.266426 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.266435 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.266472 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.266586 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.266587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.266702 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.266723 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.266724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.267351 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.267352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.267797 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.267798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.268168 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.268169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.268212 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.268593 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.268598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.269632 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.270932 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.270933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.270933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.271098 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.271315 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.271316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.271317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.271367 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.271612 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.271613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.271647 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.271710 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.271797 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 15:15:34.271960 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.266437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.8.43.188:39137)
0925 15:15:34.266466 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.266468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.266476 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:34.266527 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.266527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.266535 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.266575 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.266790 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.267281 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.267891 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.268093 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.268169 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.269721 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.269722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.269722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.269729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.270841 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.271179 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.271180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.271180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.271266 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.271432 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.271433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.271434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.271533 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.271619 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.271734 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.271847 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":15}
I20260925 15:15:34.273238 15179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Bootstrap starting.
I20260925 15:15:34.273998 15179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:34.274230 15179 log.cc:824] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:34.274894 15179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: No bootstrap required, opened a new log
I20260925 15:15:34.276321 15179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39137 } }
I20260925 15:15:34.276446 15179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:34.276467 15179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9720f9afeca241b6a20e6489603d4d25, State: Initialized, Role: FOLLOWER
I20260925 15:15:34.276584 15179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39137 } }
I20260925 15:15:34.276841 15184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39137 } } }
I20260925 15:15:34.276911 15179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:34.276921 15184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:15:34.278410 15195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:15:34.278468 15195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:15:34.280515 15111 master_runner.cc:428] Detected that this master 9720f9afeca241b6a20e6489603d4d25 is joining an existing cluster
I20260925 15:15:34.280555 15111 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.43.188:39137
I20260925 15:15:34.307544 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.282221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.8.43.188:39137)
0925 15:15:34.282251 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.282252 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.282924 (+ 672us) server_negotiation.cc:407] Connection header received
0925 15:15:34.283023 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.283024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.283031 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.283055 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.283442 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.284110 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.285073 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.285316 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.285351 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.304485 (+ 19134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.304487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.304487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.304494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.306184 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.306613 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.306614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.306615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.306662 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.306880 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.306880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.306881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.307018 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.307064 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.307329 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.307421 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:15:34.307833 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.282075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39137 (local address 127.0.0.1:54984)
0925 15:15:34.282880 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.282889 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.282941 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.283099 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.283100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.283328 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.283370 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.283372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.284201 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.284202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.284997 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.284999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.285412 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.285412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.285473 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.303821 (+ 18348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.303828 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.304388 (+ 560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.306289 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.306290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.306290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.306530 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.306737 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.306738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.306738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.306802 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.307111 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.307112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.307164 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.307644 (+ 480us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.307739 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":61,"threads_started":1}
I20260925 15:15:34.308043 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.281848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.8.43.190:34445)
0925 15:15:34.281892 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.281894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.282354 (+ 460us) server_negotiation.cc:407] Connection header received
0925 15:15:34.282444 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.282445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.282456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.282486 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.282844 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.283681 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.284722 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.285022 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.285061 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.304533 (+ 19472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.304535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.304535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.304543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.306386 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.306780 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.306781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.306782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.306836 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.307065 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.307066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.307067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.307200 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.307292 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.307517 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.307597 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:15:34.308391 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.281938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33164)
0925 15:15:34.282289 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.282297 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.282377 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.282545 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.282546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.282714 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.282766 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.282767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.283763 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.283764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.284635 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.284636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.285127 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.285127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.285185 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.285875 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.285878 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.304342 (+ 18464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.306479 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.306480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.306480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.306690 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.306916 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.306917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.306917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.306981 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.307343 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.307343 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.307391 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.308170 (+ 779us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.308267 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":76,"threads_started":1}
I20260925 15:15:34.308915 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.281841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.0.0.1:35926)
0925 15:15:34.282081 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.282092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.282164 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.282303 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.282304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.282532 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.282602 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.282604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.283536 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.283537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.284462 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.284464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.285013 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.285014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.285077 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.285547 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.285552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.304284 (+ 18732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.307041 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.307042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.307043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.307271 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.307532 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.307533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.307533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.307589 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.308385 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.308385 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.308632 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.308712 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.308811 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260925 15:15:34.308979 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.281879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.8.43.189:42655)
0925 15:15:34.281922 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.281923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.282159 (+ 236us) server_negotiation.cc:407] Connection header received
0925 15:15:34.282228 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.282228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.282239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.282298 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.282677 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.283445 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.284576 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.284890 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.284931 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.304402 (+ 19471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.304404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.304404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.304411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.306940 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.307368 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.307369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.307370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.307423 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.307798 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.307799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.307799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.307927 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.307966 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.308779 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.308871 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:15:34.310462 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 14521
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349333, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.190@KRBTEST.COM for oryx/127.8.43.188@KRBTEST.COM
I20260925 15:15:34.316718 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:38371
--local_ip_for_outbound_sockets=127.8.43.129
--tserver_master_addrs=127.8.43.190:34445
--webserver_port=34033
--webserver_interface=127.8.43.129
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab
--principal=oryx/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:15:34.319873 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.310690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39137 (local address 127.0.0.1:54988)
0925 15:15:34.311096 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.311107 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.311150 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.311883 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.311884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.312054 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.312098 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.312100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.313001 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.313003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.314010 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.314013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.314484 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.314486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.314564 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.315002 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.315008 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.316753 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.318559 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.318561 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.318561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.318738 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.319048 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.319048 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.319049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.319141 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.319523 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.319536 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.319582 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.319656 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.319768 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389}
I20260925 15:15:34.319921 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.311688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.8.43.188:39137)
0925 15:15:34.311725 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.311727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.311735 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:34.311766 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.311767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.311776 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.311832 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.312183 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.312912 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.314092 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.314398 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.314471 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.316864 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.316866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.316866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.316874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.318456 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.318861 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.318863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.318863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.318944 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.319276 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.319277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.319277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.319409 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.319487 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.319707 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.319806 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:34.320361 14459 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.43.188:39137: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9720f9afeca241b6a20e6489603d4d25"
member_type: NON_VOTER
last_known_addr {
host: "127.8.43.188"
port: 39137
}
attrs {
promote: true
}
}
dest_uuid: "743e6dbca0ce4f3da82a3d4d3371b902"
cas_config_opid_index: 5
I20260925 15:15:34.320657 14459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:34.321038 15019 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9720f9afeca241b6a20e6489603d4d25 (127.8.43.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.321285 15153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:34.322134 15019 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.43.188:39137
I20260925 15:15:34.322212 15019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.322291 15019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:34.322592 15111 master.cc:537] Master@127.8.43.188:39137 shutting down...
I20260925 15:15:34.323699 15153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 1 FOLLOWER]: Refusing update from remote peer 743e6dbca0ce4f3da82a3d4d3371b902: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:15:34.328876 15111 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:34.328975 15111 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:34.329032 15111 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: stopping tablet replica
I20260925 15:15:34.329607 15018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:15:34.331877 14443 consensus_peers.cc:487] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 -> Peer 9720f9afeca241b6a20e6489603d4d25 (127.8.43.188:39137): Couldn't send request to peer 9720f9afeca241b6a20e6489603d4d25. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:15:34.332130 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.321252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.8.43.189:42655)
0925 15:15:34.321293 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.321295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.321461 (+ 166us) server_negotiation.cc:407] Connection header received
0925 15:15:34.321488 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.321489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.321513 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.321588 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:34.321768 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.322492 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.330749 (+ 8257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.331392 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.331472 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.331576 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.332021 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.332065 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:34.332285 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.321157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.0.0.1:35934)
0925 15:15:34.321322 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.321333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.321383 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.321650 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.321651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.321653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:34.321689 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.321690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.322610 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.322611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.330632 (+ 8021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.330636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.331612 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.331613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.331679 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.331706 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.331768 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.331832 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260925 15:15:34.332562 15067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 LEARNER]: Refusing update from remote peer 743e6dbca0ce4f3da82a3d4d3371b902: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:15:34.332762 15018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 15:15:34.333914 15067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9720f9afeca241b6a20e6489603d4d25 (127.8.43.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.335983 15206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.336158 15206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:15:34.336211 15096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743e6dbca0ce4f3da82a3d4d3371b902. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: NON_VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: true } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.336294 15096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:15:34.337751 15111 master.cc:552] Master@127.8.43.188:39137 shutdown complete.
I20260925 15:15:34.337805 15111 master_runner.cc:305] Clearing existing system tablet
I20260925 15:15:34.337968 15111 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:34.338025 15111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:34.338836 15111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:15:34.339547 15208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.339787 15111 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:34.339851 15111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:15:34.339884 15111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
uuid: "9720f9afeca241b6a20e6489603d4d25"
format_stamp: "Formatted at 2026-09-25 15:15:34 on dist-test-slave-c5d6"
server_key: "ff94dc3c065a163f7322b00e43f715a7"
server_key_iv: "524590279ce177c8a64a04d6fbfe7bf2"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.340534 15111 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:34.341715 15111 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:15:34.341807 15111 log.cc:1196] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 15:15:34.342013 15111 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Deleting consensus metadata
I20260925 15:15:34.342165 15111 master_runner.cc:315] Copying system tablet from 127.8.43.190:34445
I20260925 15:15:34.342859 15111 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.43.190:34445
I20260925 15:15:34.347808 15018 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: attempting to promote NON_VOTER 863d08373b7a4f6bb257e32b07b66d89 to VOTER
I20260925 15:15:34.348073 15018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:34.348913 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.348749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39137 (local address 127.0.0.1:55000)
0925 15:15:34.348832 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.348880 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 15:15:34.349107 15067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 LEARNER]: Refusing update from remote peer 743e6dbca0ce4f3da82a3d4d3371b902: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 15:15:34.349386 15019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 15:15:34.349740 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.349611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39137 (local address 127.0.0.1:55010)
0925 15:15:34.349683 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.349709 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 15:15:34.350059 14443 consensus_peers.cc:487] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 -> Peer 9720f9afeca241b6a20e6489603d4d25 (127.8.43.188:39137): Couldn't send request to peer 9720f9afeca241b6a20e6489603d4d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:15:34.350325 15019 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 863d08373b7a4f6bb257e32b07b66d89 (127.8.43.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.350940 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.343286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.8.43.190:34445)
0925 15:15:34.343332 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.343334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.343592 (+ 258us) server_negotiation.cc:407] Connection header received
0925 15:15:34.343681 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.343682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.343695 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.343758 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.344091 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.344601 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.345434 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.345655 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.345748 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.347255 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.347256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.347257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.347265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.349291 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.349921 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.349922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.349923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.349997 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.350295 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.350296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.350297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.350432 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.350486 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.350773 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.350850 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:15:34.351466 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.343197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33170)
0925 15:15:34.343508 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.343519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.343609 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.343763 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.343764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.343961 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.344006 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.344008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.344703 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.344704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.345328 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.345330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.345748 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.345749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.345814 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.346817 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.346822 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.347130 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.349519 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.349520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.349521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.349757 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.350084 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.350084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.350085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.350208 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.350602 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.350603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.350652 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.351187 (+ 535us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.351343 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":269,"threads_started":1}
I20260925 15:15:34.351591 15067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 863d08373b7a4f6bb257e32b07b66d89 (127.8.43.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.352739 15096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.352857 15096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:15:34.353088 14479 tablet_copy_service.cc:140] P 743e6dbca0ce4f3da82a3d4d3371b902: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9720f9afeca241b6a20e6489603d4d25 ({username='oryx', principal='oryx/127.8.43.188@KRBTEST.COM'} at 127.0.0.1:33170)
I20260925 15:15:34.353164 14479 tablet_copy_service.cc:161] P 743e6dbca0ce4f3da82a3d4d3371b902: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9720f9afeca241b6a20e6489603d4d25 at {username='oryx', principal='oryx/127.8.43.188@KRBTEST.COM'} at 127.0.0.1:33170: session id = 9720f9afeca241b6a20e6489603d4d25-00000000000000000000000000000000
I20260925 15:15:34.353296 15019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.353368 15019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:34.353442 15019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.353488 15019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902 [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:34.353770 14479 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:15:34.354233 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:15:34.355175 15111 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: tablet copy: Starting download of 0 data blocks...
I20260925 15:15:34.355371 15111 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: tablet copy: Starting download of 1 WAL segments...
I20260925 15:15:34.356083 15111 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:15:34.356822 14479 tablet_copy_service.cc:342] P 743e6dbca0ce4f3da82a3d4d3371b902: Request end of tablet copy session 9720f9afeca241b6a20e6489603d4d25-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.43.188@KRBTEST.COM'} at 127.0.0.1:33170
I20260925 15:15:34.356877 14479 tablet_copy_service.cc:433] P 743e6dbca0ce4f3da82a3d4d3371b902: ending tablet copy session 9720f9afeca241b6a20e6489603d4d25-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9720f9afeca241b6a20e6489603d4d25
I20260925 15:15:34.359324 15111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:34.360535 15217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.360739 15220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:34.360721 15111 server_base.cc:1034] running on GCE node
W20260925 15:15:34.367437 15218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:34.367633 15111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:34.367797 15111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:34.368921 15111 hybrid_clock.cc:648] HybridClock initialized: now 1790349334368918 us; error 30 us; skew 500 ppm
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:34.370235 15111 init.cc:377] Logged in from keytab as oryx/127.8.43.188@KRBTEST.COM (short username oryx)
I20260925 15:15:34.370520 15111 webserver.cc:460] Webserver started at http://127.8.43.188:42371/ using document root <none> and password file <none>
I20260925 15:15:34.370637 15111 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:34.370695 15111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:34.371351 15111 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:34.371879 15227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.372035 15111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:34.372081 15111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
uuid: "9720f9afeca241b6a20e6489603d4d25"
format_stamp: "Formatted at 2026-09-25 15:15:34 on dist-test-slave-c5d6"
server_key: "ff94dc3c065a163f7322b00e43f715a7"
server_key_iv: "524590279ce177c8a64a04d6fbfe7bf2"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.372145 15111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:34.386766 15111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:34.388495 15111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:34.391616 15111 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.188:39137
I20260925 15:15:34.391969 15111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
I20260925 15:15:34.392239 15279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.188:39137 every 8 connection(s)
I20260925 15:15:34.392858 15280 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:15:34.393332 15280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Bootstrap starting.
I20260925 15:15:34.395689 15280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:15:34.395917 15280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: Bootstrap complete.
I20260925 15:15:34.396121 15280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:34.396210 15280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9720f9afeca241b6a20e6489603d4d25, State: Initialized, Role: LEARNER
I20260925 15:15:34.396255 15280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:34.396414 15280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:34.398312 15294 catalog_manager.cc:1259] Loaded cluster ID: 44c0c811054c47b9a2e16757b2d002f6
I20260925 15:15:34.398556 15294 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: loading cluster ID for follower catalog manager: success
I20260925 15:15:34.399461 15294 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: acquiring CA information for follower catalog manager: success
I20260925 15:15:34.399732 15294 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:15:34.397912 15283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } } }
I20260925 15:15:34.399814 15283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9720f9afeca241b6a20e6489603d4d25 [sys.catalog]: This master's current role is: LEARNER
W20260925 15:15:34.436579 15204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:34.436792 15204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:34.436818 15204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:34.437997 15204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:34.438041 15204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:34.438069 15204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:15:34.439545 15204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab
--principal=oryx/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:38371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=34033
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:34.439798 15204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:34.440052 15204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:34.445044 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.445111 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.445053 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:34.445536 15204 server_base.cc:1034] running on GCE node
I20260925 15:15:34.445724 15204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:34.445968 15204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:34.447130 15204 hybrid_clock.cc:648] HybridClock initialized: now 1790349334447111 us; error 32 us; skew 500 ppm
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:34.450124 15204 init.cc:377] Logged in from keytab as oryx/127.8.43.129@KRBTEST.COM (short username oryx)
I20260925 15:15:34.450611 15204 webserver.cc:460] Webserver started at http://127.8.43.129:34033/ using document root <none> and password file <none>
I20260925 15:15:34.450839 15204 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:34.450920 15204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:34.452455 15204 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:34.453100 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.453258 15204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:34.453353 15204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "0c0cc0eaf40745c3b1301f97cf484511"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "b85383f2ca4da20eb471ef32e3c43cde"
server_key_iv: "106ce0af9468111fbb2dfc641d964f00"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.453706 15204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:34.464051 15204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:34.466471 15204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:34.466642 15204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:34.466889 15204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:34.467202 15204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:34.467281 15204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.467321 15204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:34.467351 15204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.473018 15204 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:38371
I20260925 15:15:34.473073 15422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:38371 every 8 connection(s)
I20260925 15:15:34.473426 15204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:34.474027 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 15204
I20260925 15:15:34.474180 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 14655
I20260925 15:15:34.479894 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:39207
--local_ip_for_outbound_sockets=127.8.43.130
--tserver_master_addrs=127.8.43.190:34445
--webserver_port=34619
--webserver_interface=127.8.43.130
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab
--principal=oryx/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.189@KRBTEST.COM
I20260925 15:15:34.484927 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.474486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:58765 (local address 127.8.43.190:34445)
0925 15:15:34.474594 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.474596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.474937 (+ 341us) server_negotiation.cc:407] Connection header received
0925 15:15:34.475742 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.475743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.475754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.475916 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.476853 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.477365 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.478368 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.478583 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.478660 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.481217 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.481219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.481219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.481226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.482881 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.483424 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.483425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.483425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.483486 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.483792 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.483793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.483794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.484225 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.484291 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.484673 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.484786 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.188@KRBTEST.COM
I20260925 15:15:34.486300 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.474155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:58765)
0925 15:15:34.474794 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.474840 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.475629 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.476024 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.476027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.476567 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.476753 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.476757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.477457 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.477458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.478275 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.478277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.478667 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.478668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.478784 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.479296 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.479306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.481114 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.482964 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.482970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.482972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.483329 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.483557 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.483557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.483558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.483708 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.484330 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.484337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.484506 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.485069 (+ 563us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.485537 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":77,"threads_started":1}
I20260925 15:15:34.487017 15424 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34445
I20260925 15:15:34.487120 15424 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.487349 15424 heartbeater.cc:511] Master 127.8.43.190:34445 requested a full tablet report, sending...
I20260925 15:15:34.487751 14459 ts_manager.cc:194] Re-registered known tserver with Master: 0c0cc0eaf40745c3b1301f97cf484511 (127.8.43.129:38371)
I20260925 15:15:34.488278 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.474439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:53783 (local address 127.8.43.189:42655)
0925 15:15:34.474536 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.474538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.479501 (+ 4963us) server_negotiation.cc:407] Connection header received
0925 15:15:34.479553 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.479554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.479566 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.479622 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.480101 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.480836 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.481932 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.482219 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.482261 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.484888 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.484889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.484890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.484898 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.486615 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.487449 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.487450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.487451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.487510 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.487709 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.487709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.487710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.487843 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.487893 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.488119 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.488195 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:15:34.488276 14459 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:58765
I20260925 15:15:34.488421 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.474635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:50359 (local address 127.8.43.188:39137)
0925 15:15:34.474767 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.474770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.479637 (+ 4867us) server_negotiation.cc:407] Connection header received
0925 15:15:34.479727 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.479729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.479742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.479779 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.480511 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.481810 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.483259 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.483551 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.483614 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.485681 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.485682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.485683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.485690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.487316 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.487665 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.487667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.487667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.487715 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.487895 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.487896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.487896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.488021 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.488066 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.488242 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.488320 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260925 15:15:34.488729 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.474435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39137 (local address 127.8.43.129:50359)
0925 15:15:34.479588 (+ 5153us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.479596 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.479661 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.479845 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.479846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.480309 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.480393 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.480395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.481906 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.481907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.483148 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.483150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.483668 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.483669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.483752 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.484163 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.484169 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.485578 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.487396 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.487396 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.487397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.487592 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.487783 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.487784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.487784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.487828 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.488110 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.488110 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.488140 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.488533 (+ 393us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.488634 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5140,"thread_start_us":49,"threads_started":1}
I20260925 15:15:34.489117 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.474156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.8.43.129:53783)
0925 15:15:34.479411 (+ 5255us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.479423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.479488 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.479672 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.479673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.479929 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.480005 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.480006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.480930 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.480930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.481830 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.481832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.482340 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.482340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.482384 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.483301 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.483305 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.484791 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.487003 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.487004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.487004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.487319 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.487587 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.487587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.487588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.487642 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.487974 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.487975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.488009 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.488750 (+ 741us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.489045 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5228,"mutex_wait_us":40,"thread_start_us":79,"threads_started":1}
I20260925 15:15:34.489459 15425 heartbeater.cc:347] Connected to a master server at 127.8.43.189:42655
I20260925 15:15:34.489509 15425 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.489583 15425 heartbeater.cc:511] Master 127.8.43.189:42655 requested a full tablet report, sending...
I20260925 15:15:34.489657 15423 heartbeater.cc:347] Connected to a master server at 127.8.43.188:39137
I20260925 15:15:34.489706 15423 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.489770 15423 heartbeater.cc:511] Master 127.8.43.188:39137 requested a full tablet report, sending...
I20260925 15:15:34.489914 15057 ts_manager.cc:194] Registered new tserver with Master: 0c0cc0eaf40745c3b1301f97cf484511 (127.8.43.129:38371)
I20260925 15:15:34.490307 15244 ts_manager.cc:194] Registered new tserver with Master: 0c0cc0eaf40745c3b1301f97cf484511 (127.8.43.129:38371)
W20260925 15:15:34.565470 15431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:34.565672 15431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:34.565696 15431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:34.566883 15431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:34.566926 15431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:34.566979 15431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:15:34.568495 15431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab
--principal=oryx/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:39207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=34619
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:34.568748 15431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:34.568981 15431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:34.573900 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.573922 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.573900 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:34.573925 15431 server_base.cc:1034] running on GCE node
I20260925 15:15:34.574345 15431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:34.574635 15431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:34.575793 15431 hybrid_clock.cc:648] HybridClock initialized: now 1790349334575777 us; error 42 us; skew 500 ppm
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:34.578186 15431 init.cc:377] Logged in from keytab as oryx/127.8.43.130@KRBTEST.COM (short username oryx)
I20260925 15:15:34.578593 15431 webserver.cc:460] Webserver started at http://127.8.43.130:34619/ using document root <none> and password file <none>
I20260925 15:15:34.578825 15431 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:34.578905 15431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:34.580256 15431 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:34.580825 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.581012 15431 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:34.581108 15431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "aaaf0e4f20874f49b5a970a66e72f0b0"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "196a40a723132381e1c95bb4a01b5892"
server_key_iv: "9b6d046e1374db1449810ee34aa76435"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.581465 15431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:34.614705 15431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:34.616812 15431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:34.616967 15431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:34.617235 15431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:34.617548 15431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:34.617580 15431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.617623 15431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:34.617650 15431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.623405 15431 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:39207
I20260925 15:15:34.623509 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:39207 every 8 connection(s)
I20260925 15:15:34.623837 15431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:34.625777 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 15431
I20260925 15:15:34.625950 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 14789
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.189@KRBTEST.COM
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:15:34.633770 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:42813
--local_ip_for_outbound_sockets=127.8.43.131
--tserver_master_addrs=127.8.43.190:34445
--webserver_port=43017
--webserver_interface=127.8.43.131
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab
--principal=oryx/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.188@KRBTEST.COM
I20260925 15:15:34.638399 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.624970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:37411 (local address 127.8.43.189:42655)
0925 15:15:34.625030 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.625032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.625101 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:15:34.626309 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.626311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.626324 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.626354 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.627410 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.628234 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.629913 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.630273 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.630318 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.634763 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.634764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.634764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.634773 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.636702 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637242 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.637243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.637244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.637303 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637578 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.637579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.637579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.637724 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.637765 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.638175 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.638262 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:15:34.638549 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.624603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:45223 (local address 127.8.43.188:39137)
0925 15:15:34.624648 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.624650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.624891 (+ 241us) server_negotiation.cc:407] Connection header received
0925 15:15:34.627460 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.627462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.627479 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.627510 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.628027 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.628776 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.629921 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.630235 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.630269 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.634763 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.634764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.634764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.634773 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.636995 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637521 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.637522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.637523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.637581 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637809 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.637810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.637810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.638158 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.638215 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.638385 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.638471 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:34.638608 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.625083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:53609)
0925 15:15:34.625186 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.625193 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.627275 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.627519 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.627520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.627827 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.627904 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.627906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.628877 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.628878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.629814 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.629816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.630349 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.630351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.630503 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.631358 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.631362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.634669 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.637092 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.637093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.637450 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.637653 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.637654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.637745 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.638033 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.638034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.638076 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.638150 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.638344 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":46,"thread_start_us":67,"threads_started":1}
I20260925 15:15:34.638978 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.624880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.8.43.130:37411)
0925 15:15:34.625033 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.625044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.626184 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.626543 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.626548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.626966 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.627301 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.627308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.628361 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.628362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.629815 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.629818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.630395 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.630396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.630501 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.631197 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.631209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.634669 (+ 3460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.636798 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.636804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.636807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.637128 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.637371 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.637372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.637507 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.637809 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.637814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.637993 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.638532 (+ 539us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.638656 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260925 15:15:34.639185 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.624593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39137 (local address 127.8.43.130:45223)
0925 15:15:34.624750 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.624793 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.627373 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.627572 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.627572 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.627823 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.627926 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.627928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.628877 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.628878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.629821 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.629822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.630351 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.630351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.630502 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.631273 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.631278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.634644 (+ 3366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.637092 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.637093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.637415 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.637654 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.637655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.637733 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.638241 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.638242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.638275 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.638661 (+ 386us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.638802 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":40,"thread_start_us":85,"threads_started":1}
I20260925 15:15:34.639425 15562 heartbeater.cc:347] Connected to a master server at 127.8.43.189:42655
I20260925 15:15:34.639523 15562 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.639520 15560 heartbeater.cc:347] Connected to a master server at 127.8.43.188:39137
I20260925 15:15:34.639598 15560 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.639742 15562 heartbeater.cc:511] Master 127.8.43.189:42655 requested a full tablet report, sending...
I20260925 15:15:34.639745 15560 heartbeater.cc:511] Master 127.8.43.188:39137 requested a full tablet report, sending...
I20260925 15:15:34.639918 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.625260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:53609 (local address 127.8.43.190:34445)
0925 15:15:34.625302 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.625304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.625312 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:34.627375 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.627377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.627390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.627443 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.628009 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.628785 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.629920 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.630211 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.630253 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.634772 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.634773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.634774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.634782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.637004 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637530 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.637531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.637531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.637581 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.637811 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.637812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.637812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.637945 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.638029 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.639748 (+ 1719us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.639833 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:34.640192 15561 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34445
I20260925 15:15:34.640213 15244 ts_manager.cc:194] Registered new tserver with Master: aaaf0e4f20874f49b5a970a66e72f0b0 (127.8.43.130:39207)
I20260925 15:15:34.640233 15561 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.640223 15057 ts_manager.cc:194] Registered new tserver with Master: aaaf0e4f20874f49b5a970a66e72f0b0 (127.8.43.130:39207)
I20260925 15:15:34.640292 15561 heartbeater.cc:511] Master 127.8.43.190:34445 requested a full tablet report, sending...
I20260925 15:15:34.640501 14459 ts_manager.cc:194] Re-registered known tserver with Master: aaaf0e4f20874f49b5a970a66e72f0b0 (127.8.43.130:39207)
I20260925 15:15:34.640849 14459 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:53609
W20260925 15:15:34.717902 15567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:34.718112 15567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:34.718135 15567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:34.719205 15567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:34.719270 15567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:34.719296 15567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:15:34.720741 15567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab
--principal=oryx/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:42813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=43017
--tserver_master_addrs=127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:15:34.720974 15567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:34.721206 15567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:34.726217 15573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:34.726227 15567 server_base.cc:1034] running on GCE node
W20260925 15:15:34.726187 15572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:34.726198 15575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:34.726615 15567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:34.726889 15567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:34.728044 15567 hybrid_clock.cc:648] HybridClock initialized: now 1790349334728033 us; error 29 us; skew 500 ppm
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:34.730461 15567 init.cc:377] Logged in from keytab as oryx/127.8.43.131@KRBTEST.COM (short username oryx)
I20260925 15:15:34.730909 15567 webserver.cc:460] Webserver started at http://127.8.43.131:43017/ using document root <none> and password file <none>
I20260925 15:15:34.731130 15567 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:34.731208 15567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:34.732532 15567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:15:34.733079 15582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.733233 15567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:15:34.733331 15567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "c1c7eaab79f14061bdf0381a5778406e"
format_stamp: "Formatted at 2026-09-25 15:15:33 on dist-test-slave-c5d6"
server_key: "cce693c07aa3f5235235a2adcb242dd4"
server_key_iv: "1b6f4144b5cdd6c951d4ff3028835949"
server_key_version: "encryptionkey@0"
I20260925 15:15:34.733678 15567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:34.753808 15567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:34.755908 15567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:34.756101 15567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:34.756350 15567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:34.756678 15567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:34.756721 15567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.756752 15567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:34.756778 15567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:34.762758 15567 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:42813
I20260925 15:15:34.762841 15695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:42813 every 8 connection(s)
I20260925 15:15:34.763190 15567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.188@KRBTEST.COM
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:15:34.770170 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 15567
Sep 25 15:15:34 dist-test-slave-c5d6 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349334, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.189@KRBTEST.COM
I20260925 15:15:34.775719 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.763919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:45331 (local address 127.8.43.190:34445)
0925 15:15:34.764319 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.764321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.764330 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:34.764800 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.764801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.764815 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.764878 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.765560 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.766338 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.767793 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.767971 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.768026 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.772246 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.772248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.772248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.772257 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.774203 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.774660 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.774661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.774662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.774722 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.774968 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.774968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.774969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.775100 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.775145 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.775530 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.775613 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368}
I20260925 15:15:34.776151 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.763922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:38975 (local address 127.8.43.189:42655)
0925 15:15:34.763968 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.763970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.764197 (+ 227us) server_negotiation.cc:407] Connection header received
0925 15:15:34.764819 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.764820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.764833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.764866 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.765625 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.766352 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.767210 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.767528 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.767593 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.771879 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.771881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.771881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.771890 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.774340 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.774744 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.774745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.774746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.774803 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.775271 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.775272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.775272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.775403 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.775448 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.775975 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.776060 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:34.776571 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.763923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:45331)
0925 15:15:34.764080 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.764119 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.764701 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.764945 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.764949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.765281 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.765466 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.765470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.766422 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.766423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.767320 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.767323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.768041 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.768042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.768112 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.768710 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.768716 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.771470 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.774280 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.774281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.774281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.774570 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.774787 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.774788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.774788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.774904 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.775191 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.775196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.775372 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.776112 (+ 740us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.776297 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260925 15:15:34.777171 15697 heartbeater.cc:347] Connected to a master server at 127.8.43.190:34445
I20260925 15:15:34.777292 15697 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.777506 15697 heartbeater.cc:511] Master 127.8.43.190:34445 requested a full tablet report, sending...
I20260925 15:15:34.777874 14459 ts_manager.cc:194] Re-registered known tserver with Master: c1c7eaab79f14061bdf0381a5778406e (127.8.43.131:42813)
I20260925 15:15:34.778381 14459 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:45331
I20260925 15:15:34.779728 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.763923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42655 (local address 127.8.43.131:38975)
0925 15:15:34.764107 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.764122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.764726 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.765318 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.765319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.765464 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.765528 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.765530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.766437 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.766438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.767112 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.767113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.767622 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.767623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.767722 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.768851 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.768856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.771768 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.774437 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.774438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.774439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.774655 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.775103 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.775104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.775104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.775167 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.775498 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.775499 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.775540 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.779467 (+ 3927us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.779587 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":28,"thread_start_us":122,"threads_started":1}
I20260925 15:15:34.780159 15698 heartbeater.cc:347] Connected to a master server at 127.8.43.189:42655
I20260925 15:15:34.780212 15698 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.780278 15698 heartbeater.cc:511] Master 127.8.43.189:42655 requested a full tablet report, sending...
I20260925 15:15:34.780483 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.764271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:56311 (local address 127.8.43.188:39137)
0925 15:15:34.764314 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.764316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.764456 (+ 140us) server_negotiation.cc:407] Connection header received
0925 15:15:34.764819 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.764820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.764833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.764866 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.765577 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.766095 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.767430 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.767704 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.767753 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.771171 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.771172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.771173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.771180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.773030 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.775643 (+ 2613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.775644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.775645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.775708 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.779866 (+ 4158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.779868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.779868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.780019 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.780070 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.780304 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.780380 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:15:34.780560 15057 ts_manager.cc:194] Registered new tserver with Master: c1c7eaab79f14061bdf0381a5778406e (127.8.43.131:42813)
I20260925 15:15:34.780603 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.771462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.8.43.190:34445)
0925 15:15:34.772083 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.772085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.772100 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:15:34.772349 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.772350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.772364 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.772431 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.773300 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.774122 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.775153 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.775449 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.775520 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.777004 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.777005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.777006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.777012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.778846 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.779555 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.779556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.779556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.779616 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.779832 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.779833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.779834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.779970 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.780075 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.780441 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.780517 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":105,"threads_started":1}
I20260925 15:15:34.781059 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.764202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39137 (local address 127.8.43.131:56311)
0925 15:15:34.764391 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.764399 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.764721 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.764962 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.764963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.765292 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:34.765468 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.765473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.766211 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.766212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.767325 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.767326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.767785 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.767786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.767843 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.768441 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:34.768451 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:34.771045 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:34.773487 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.773495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.773497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.775007 (+ 1510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.775790 (+ 783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:34.775790 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:34.775791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:34.775859 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:34.780132 (+ 4273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:34.780133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:34.780178 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.780592 (+ 414us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.780712 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":29,"thread_start_us":57,"threads_started":1}
I20260925 15:15:34.781384 15696 heartbeater.cc:347] Connected to a master server at 127.8.43.188:39137
I20260925 15:15:34.781446 15696 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:34.781508 15696 heartbeater.cc:511] Master 127.8.43.188:39137 requested a full tablet report, sending...
I20260925 15:15:34.781805 15244 ts_manager.cc:194] Registered new tserver with Master: c1c7eaab79f14061bdf0381a5778406e (127.8.43.131:42813)
I20260925 15:15:34.782850 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.771728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.8.43.188:39137)
0925 15:15:34.771863 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.771865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.772722 (+ 857us) server_negotiation.cc:407] Connection header received
0925 15:15:34.772789 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.772789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.772803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.772861 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.773863 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.774615 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.775711 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.776476 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.776547 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.778617 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.778618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.778618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.778625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.780709 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.781612 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.781613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.781614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.781677 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.781904 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.781905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.781905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.782155 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.782254 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.782658 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.782743 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260925 15:15:34.782310 14459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:34.785010 14459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:34.785207 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.771955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.8.43.189:42655)
0925 15:15:34.776235 (+ 4280us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.776237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.776247 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:15:34.776268 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.776269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.776282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.776341 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.777073 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.777754 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.778835 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.779117 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.779162 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.782397 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.782398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.782398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.782404 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.784029 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.784381 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.784382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.784383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.784441 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.784681 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.784682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.784682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.784810 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.784904 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.785062 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.785130 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4263,"thread_start_us":479,"threads_started":1}
I20260925 15:15:34.793130 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.789254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:39207 (local address 127.0.0.1:40862)
0925 15:15:34.789478 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.789487 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.789532 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.790066 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.790068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.790071 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:34.790130 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.790132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.791399 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.791399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.792222 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.792224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.792890 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.792891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.792947 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.792952 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.793014 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.793052 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":40,"thread_start_us":77,"threads_started":1}
I20260925 15:15:34.793179 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.789706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.8.43.130:39207)
0925 15:15:34.789878 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.789883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.789894 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:15:34.789933 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.789935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.789950 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.790038 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:34.790220 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.791326 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.792304 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.792797 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.792894 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.792976 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.793068 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.793108 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260925 15:15:34.793906 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.789203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:38371 (local address 127.0.0.1:35732)
0925 15:15:34.789445 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.789458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.789506 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.790152 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.790153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.790154 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:34.790184 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.790185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.791485 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.791486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.792669 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.792672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.793622 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.793623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.793691 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.793695 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.793758 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.793806 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260925 15:15:34.793913 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.789293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.8.43.129:38371)
0925 15:15:34.789970 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.789975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.789985 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:15:34.790027 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.790029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.790048 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.790158 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:34.790221 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.791421 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.792765 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.793509 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.793620 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.793695 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.793788 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.793832 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":93,"threads_started":1}
I20260925 15:15:34.793841 15494 tablet_service.cc:1467] Processing CreateTablet for tablet 192c5d5205234fb0906c7cf5c8dd2c76 (DEFAULT_TABLE table=test-table [id=94b319965ddf41a49f3e32ab93e6e9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:34.794209 15494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192c5d5205234fb0906c7cf5c8dd2c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:34.794641 15357 tablet_service.cc:1467] Processing CreateTablet for tablet 192c5d5205234fb0906c7cf5c8dd2c76 (DEFAULT_TABLE table=test-table [id=94b319965ddf41a49f3e32ab93e6e9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:34.794965 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192c5d5205234fb0906c7cf5c8dd2c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:34.797070 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.789304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:42813 (local address 127.0.0.1:41012)
0925 15:15:34.789613 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.789621 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.791274 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.791730 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.791731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.791733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:34.791767 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.791768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.794793 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.794794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.795938 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.795941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.796810 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.796811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.796875 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.796880 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.796943 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.796988 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":86,"threads_started":1}
I20260925 15:15:34.797217 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.789687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.8.43.131:42813)
0925 15:15:34.791557 (+ 1870us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.791561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.791570 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:15:34.791603 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.791605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.791619 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.791721 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:34.791830 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.794688 (+ 2858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.796030 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.796718 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.796787 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.796940 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.796968 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.797030 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1838,"thread_start_us":73,"threads_started":1}
I20260925 15:15:34.797787 15722 tablet_bootstrap.cc:492] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: Bootstrap starting.
I20260925 15:15:34.797923 15630 tablet_service.cc:1467] Processing CreateTablet for tablet 192c5d5205234fb0906c7cf5c8dd2c76 (DEFAULT_TABLE table=test-table [id=94b319965ddf41a49f3e32ab93e6e9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:34.798197 15630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192c5d5205234fb0906c7cf5c8dd2c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:34.798512 15722 tablet_bootstrap.cc:654] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:34.798731 15723 tablet_bootstrap.cc:492] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: Bootstrap starting.
I20260925 15:15:34.798866 15722 log.cc:824] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:34.799393 15723 tablet_bootstrap.cc:654] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:34.799700 15722 tablet_bootstrap.cc:492] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: No bootstrap required, opened a new log
I20260925 15:15:34.799780 15722 ts_tablet_manager.cc:1397] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:34.799784 15723 log.cc:824] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:34.800953 15726 tablet_bootstrap.cc:492] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: Bootstrap starting.
I20260925 15:15:34.801620 15726 tablet_bootstrap.cc:654] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:34.801926 15726 log.cc:824] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:34.802306 15722 raft_consensus.cc:348] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.802508 15722 raft_consensus.cc:374] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:34.802551 15722 raft_consensus.cc:729] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaaf0e4f20874f49b5a970a66e72f0b0, State: Initialized, Role: FOLLOWER
I20260925 15:15:34.802695 15726 tablet_bootstrap.cc:492] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: No bootstrap required, opened a new log
I20260925 15:15:34.802671 15722 consensus_queue.cc:260] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.802760 15726 ts_tablet_manager.cc:1397] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:15:34.802953 15722 ts_tablet_manager.cc:1428] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:34.803197 15561 heartbeater.cc:503] Master 127.8.43.190:34445 was elected leader, sending a full tablet report...
I20260925 15:15:34.804062 15723 tablet_bootstrap.cc:492] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: No bootstrap required, opened a new log
I20260925 15:15:34.804136 15723 ts_tablet_manager.cc:1397] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 15:15:34.805421 15726 raft_consensus.cc:348] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.805603 15726 raft_consensus.cc:374] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:34.805641 15726 raft_consensus.cc:729] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c7eaab79f14061bdf0381a5778406e, State: Initialized, Role: FOLLOWER
I20260925 15:15:34.805763 15726 consensus_queue.cc:260] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.806025 15726 ts_tablet_manager.cc:1428] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 15:15:34.806162 15697 heartbeater.cc:503] Master 127.8.43.190:34445 was elected leader, sending a full tablet report...
I20260925 15:15:34.806915 15723 raft_consensus.cc:348] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.807088 15723 raft_consensus.cc:374] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:34.807122 15723 raft_consensus.cc:729] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0cc0eaf40745c3b1301f97cf484511, State: Initialized, Role: FOLLOWER
I20260925 15:15:34.807204 15723 consensus_queue.cc:260] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.807510 15723 ts_tablet_manager.cc:1428] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 15:15:34.807646 15424 heartbeater.cc:503] Master 127.8.43.190:34445 was elected leader, sending a full tablet report...
I20260925 15:15:34.808207 15729 raft_consensus.cc:482] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:34.808329 15729 raft_consensus.cc:504] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.808831 15729 leader_election.cc:288] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c0cc0eaf40745c3b1301f97cf484511 (127.8.43.129:38371), aaaf0e4f20874f49b5a970a66e72f0b0 (127.8.43.130:39207)
I20260925 15:15:34.811698 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.809033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:42185 (local address 127.8.43.129:38371)
0925 15:15:34.809224 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.809226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.809231 (+ 5us) server_negotiation.cc:407] Connection header received
0925 15:15:34.809246 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.809247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.809254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.809316 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:34.809445 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.809895 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.810876 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.811330 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.811425 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.811441 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.811601 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.811634 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260925 15:15:34.811704 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.808992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:38371 (local address 127.8.43.131:42185)
0925 15:15:34.809141 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.809148 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.809189 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.809320 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.809321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.809323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:34.809368 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.809370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.810007 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.810008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.810757 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.810759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.811425 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.811426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.811501 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.811504 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.811594 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.811632 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":33}
I20260925 15:15:34.811738 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.809030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:59685 (local address 127.8.43.130:39207)
0925 15:15:34.809072 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.809074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.809082 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:15:34.809128 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.809128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.809136 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.809185 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:34.809383 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.809974 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.810842 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.811371 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.811460 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.811467 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.811638 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.811670 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:15:34.811789 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.808983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:39207 (local address 127.8.43.131:59685)
0925 15:15:34.809019 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:34.809032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:34.809072 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:34.809210 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:34.809211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:34.809213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:34.809307 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.809309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.810072 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.810073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.810759 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:34.810761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.811462 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.811463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:34.811554 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.811558 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:15:34.811630 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:15:34.811670 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":1}
I20260925 15:15:34.812028 15377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192c5d5205234fb0906c7cf5c8dd2c76" candidate_uuid: "c1c7eaab79f14061bdf0381a5778406e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0cc0eaf40745c3b1301f97cf484511" is_pre_election: true
I20260925 15:15:34.812062 15514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192c5d5205234fb0906c7cf5c8dd2c76" candidate_uuid: "c1c7eaab79f14061bdf0381a5778406e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" is_pre_election: true
I20260925 15:15:34.812160 15377 raft_consensus.cc:2454] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c7eaab79f14061bdf0381a5778406e in term 0.
I20260925 15:15:34.812180 15514 raft_consensus.cc:2454] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c7eaab79f14061bdf0381a5778406e in term 0.
I20260925 15:15:34.812382 15584 leader_election.cc:302] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaaf0e4f20874f49b5a970a66e72f0b0, c1c7eaab79f14061bdf0381a5778406e; no voters:
I20260925 15:15:34.812583 15729 raft_consensus.cc:2781] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:34.812661 15729 raft_consensus.cc:482] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:34.812712 15729 raft_consensus.cc:3037] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:34.813359 15729 raft_consensus.cc:504] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.813519 15729 leader_election.cc:288] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [CANDIDATE]: Term 1 election: Requested vote from peers 0c0cc0eaf40745c3b1301f97cf484511 (127.8.43.129:38371), aaaf0e4f20874f49b5a970a66e72f0b0 (127.8.43.130:39207)
I20260925 15:15:34.813656 15514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192c5d5205234fb0906c7cf5c8dd2c76" candidate_uuid: "c1c7eaab79f14061bdf0381a5778406e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0"
I20260925 15:15:34.813704 15377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192c5d5205234fb0906c7cf5c8dd2c76" candidate_uuid: "c1c7eaab79f14061bdf0381a5778406e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0cc0eaf40745c3b1301f97cf484511"
I20260925 15:15:34.813740 15514 raft_consensus.cc:3037] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:34.813763 15377 raft_consensus.cc:3037] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:34.814610 15377 raft_consensus.cc:2454] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c7eaab79f14061bdf0381a5778406e in term 1.
I20260925 15:15:34.814610 15514 raft_consensus.cc:2454] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c7eaab79f14061bdf0381a5778406e in term 1.
I20260925 15:15:34.814782 15585 leader_election.cc:302] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0cc0eaf40745c3b1301f97cf484511, c1c7eaab79f14061bdf0381a5778406e; no voters:
I20260925 15:15:34.814870 15729 raft_consensus.cc:2781] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:34.815016 15729 raft_consensus.cc:686] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 1 LEADER]: Becoming Leader. State: Replica: c1c7eaab79f14061bdf0381a5778406e, State: Running, Role: LEADER
I20260925 15:15:34.815129 15729 consensus_queue.cc:237] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } }
I20260925 15:15:34.815989 14459 catalog_manager.cc:5495] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e reported cstate change: term changed from 0 to 1, leader changed from <none> to c1c7eaab79f14061bdf0381a5778406e (127.8.43.131). New cstate: current_term: 1 leader_uuid: "c1c7eaab79f14061bdf0381a5778406e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1c7eaab79f14061bdf0381a5778406e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 42813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 } health_report { overall_health: UNKNOWN } } }
I20260925 15:15:34.823606 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.821476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.8.43.131:42813)
0925 15:15:34.821538 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.821544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.821561 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:15:34.821586 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.821586 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.821601 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.821660 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:34.821809 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.822249 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.822935 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.823106 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.823153 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.823337 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:34.823425 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:34.823484 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.823517 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.823535 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:15:34.825316 15514 raft_consensus.cc:1261] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 1 FOLLOWER]: Refusing update from remote peer c1c7eaab79f14061bdf0381a5778406e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:34.825335 15377 raft_consensus.cc:1261] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 1 FOLLOWER]: Refusing update from remote peer c1c7eaab79f14061bdf0381a5778406e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:34.825664 15729 consensus_queue.cc:1035] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0cc0eaf40745c3b1301f97cf484511" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:34.825783 15731 consensus_queue.cc:1035] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaaf0e4f20874f49b5a970a66e72f0b0" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:34.829578 15736 mvcc.cc:205] Tried to move back new op lower bound from 7333270875441999872 to 7333270875403735040. Current Snapshot: MvccSnapshot[applied={T|T < 7333270875441999872}]
I20260925 15:15:34.830159 15735 mvcc.cc:205] Tried to move back new op lower bound from 7333270875441999872 to 7333270875403735040. Current Snapshot: MvccSnapshot[applied={T|T < 7333270875441999872}]
I20260925 15:15:34.832564 14459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:34.832657 14459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:34.833858 14459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 743e6dbca0ce4f3da82a3d4d3371b902: Sending DeleteTablet for 3 replicas of tablet 192c5d5205234fb0906c7cf5c8dd2c76
I20260925 15:15:34.834196 15630 tablet_service.cc:1514] Processing DeleteTablet for tablet 192c5d5205234fb0906c7cf5c8dd2c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:34 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:41012
I20260925 15:15:34.834570 15494 tablet_service.cc:1514] Processing DeleteTablet for tablet 192c5d5205234fb0906c7cf5c8dd2c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:34 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:40862
I20260925 15:15:34.834839 15357 tablet_service.cc:1514] Processing DeleteTablet for tablet 192c5d5205234fb0906c7cf5c8dd2c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:34 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:35732
I20260925 15:15:34.835043 15743 tablet_replica.cc:330] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: stopping tablet replica
I20260925 15:15:34.835099 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 14427
I20260925 15:15:34.835131 15743 raft_consensus.cc:2229] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:34.835235 15743 raft_consensus.cc:2258] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:34.836088 15743 ts_tablet_manager.cc:1905] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:34.836479 15745 tablet_replica.cc:330] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: stopping tablet replica
I20260925 15:15:34.836376 15746 tablet_replica.cc:330] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: stopping tablet replica
I20260925 15:15:34.836655 15745 raft_consensus.cc:2229] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:34.836768 15746 raft_consensus.cc:2229] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:34.836910 15745 raft_consensus.cc:2258] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:34.837061 15746 raft_consensus.cc:2258] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:34.837741 15745 ts_tablet_manager.cc:1905] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:34.837925 15743 ts_tablet_manager.cc:1918] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:34.837998 15743 log.cc:1196] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/192c5d5205234fb0906c7cf5c8dd2c76
I20260925 15:15:34.838256 15743 ts_tablet_manager.cc:1939] T 192c5d5205234fb0906c7cf5c8dd2c76 P c1c7eaab79f14061bdf0381a5778406e: Deleting consensus metadata
I20260925 15:15:34.839371 15745 ts_tablet_manager.cc:1918] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:34.839450 15745 log.cc:1196] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/192c5d5205234fb0906c7cf5c8dd2c76
I20260925 15:15:34.839684 15745 ts_tablet_manager.cc:1939] T 192c5d5205234fb0906c7cf5c8dd2c76 P aaaf0e4f20874f49b5a970a66e72f0b0: Deleting consensus metadata
W20260925 15:15:34.841699 15449 connection.cc:535] server connection from 127.0.0.1:40862 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 15:15:34.841699 15586 connection.cc:535] server connection from 127.0.0.1:41012 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:15:34.842079 15746 ts_tablet_manager.cc:1905] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:34.844066 15746 ts_tablet_manager.cc:1918] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:34.844130 15746 log.cc:1196] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/192c5d5205234fb0906c7cf5c8dd2c76
I20260925 15:15:34.844403 15746 ts_tablet_manager.cc:1939] T 192c5d5205234fb0906c7cf5c8dd2c76 P 0c0cc0eaf40745c3b1301f97cf484511: Deleting consensus metadata
W20260925 15:15:34.844769 15310 connection.cc:441] server connection from 127.0.0.1:35732 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35732 (request call id 1) could send its response
I20260925 15:15:34.850796 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.834547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.8.43.188:39137)
0925 15:15:34.834587 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.834589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.834737 (+ 148us) server_negotiation.cc:407] Connection header received
0925 15:15:34.834830 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.834831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.834844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.834872 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.843373 (+ 8501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.844144 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.845370 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.845678 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.845719 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.847361 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.847362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.847362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.847370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.849253 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.849668 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.849669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.849670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.849726 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.849943 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.849944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.849944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.850107 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.850214 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.850619 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.850700 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:15:34.852393 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.843908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.8.43.189:42655)
0925 15:15:34.843953 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.843955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.844649 (+ 694us) server_negotiation.cc:407] Connection header received
0925 15:15:34.844743 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.844744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.844755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.844784 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.845594 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.846311 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.847372 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.847677 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.847717 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.849451 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.849452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.849452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.849459 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.851201 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.851569 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.851570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.851571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.851625 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.851822 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.851823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.851823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.851948 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.852010 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.852212 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.852287 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:15:34.852967 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:34.844146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.8.43.188:39137)
0925 15:15:34.844184 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:34.844186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:34.845024 (+ 838us) server_negotiation.cc:407] Connection header received
0925 15:15:34.845115 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:34.845116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:34.845126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:34.845154 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:34.845787 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.846469 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.847419 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:34.847703 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:34.847738 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:34.849451 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:34.849452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:34.849453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:34.849469 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:34.851564 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.852111 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.852112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.852112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.852166 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:34.852362 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:34.852364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:34.852364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:34.852489 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:34.852587 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:34.852780 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 15:15:34.852858 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:15:35.830700 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:35.830421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:34405)
0925 15:15:35.830567 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:35.830645 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260925 15:15:35.830713 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:35.830456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:36423)
0925 15:15:35.830623 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:35.830677 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260925 15:15:35.830701 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:35.830421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:37173)
0925 15:15:35.830569 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:35.830645 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
W20260925 15:15:35.830895 15424 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:34445 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
W20260925 15:15:35.830906 15561 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:34445 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
W20260925 15:15:35.830993 15697 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:34445 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:36.334128 15757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 743e6dbca0ce4f3da82a3d4d3371b902)
I20260925 15:15:36.334245 15757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:36.334451 15757 leader_election.cc:288] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445)
I20260925 15:15:36.334765 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:36.334588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33608)
0925 15:15:36.334681 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:36.334729 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
W20260925 15:15:36.334955 15044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:36.335418 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:36.335306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33612)
0925 15:15:36.335355 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:36.335397 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:15:36.335564 15044 leader_election.cc:334] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445): Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:36.335623 15044 leader_election.cc:302] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 863d08373b7a4f6bb257e32b07b66d89; no voters: 743e6dbca0ce4f3da82a3d4d3371b902
I20260925 15:15:36.335716 15757 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:15:36.831617 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:36.831377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:36881)
0925 15:15:36.831547 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:36.831576 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260925 15:15:36.831779 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:36.831377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:43083)
0925 15:15:36.831512 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:36.831556 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":94,"threads_started":1}
I20260925 15:15:36.831804 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:36.831361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:60935)
0925 15:15:36.831490 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:36.831540 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260925 15:15:37.832682 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:37.832320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:40155)
0925 15:15:37.832545 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:37.832611 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
I20260925 15:15:37.832764 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:37.832320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:35427)
0925 15:15:37.832678 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:37.832711 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":64,"threads_started":1}
I20260925 15:15:37.832978 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:37.832443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:58213)
0925 15:15:37.832902 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:37.832937 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":65,"threads_started":1}
I20260925 15:15:38.041941 15766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 743e6dbca0ce4f3da82a3d4d3371b902)
I20260925 15:15:38.042114 15766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:38.042304 15766 leader_election.cc:288] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445)
I20260925 15:15:38.042714 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:38.042468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33830)
0925 15:15:38.042621 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:38.042678 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260925 15:15:38.043210 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:38.043109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:33838)
0925 15:15:38.043163 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:38.043192 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:15:38.043326 15044 leader_election.cc:334] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445): Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:38.043388 15044 leader_election.cc:302] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 863d08373b7a4f6bb257e32b07b66d89; no voters: 743e6dbca0ce4f3da82a3d4d3371b902
I20260925 15:15:38.043515 15766 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:15:38.833564 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:38.833268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:33477)
0925 15:15:38.833472 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:38.833508 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":50,"threads_started":1}
I20260925 15:15:38.833624 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:38.833181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:44397)
0925 15:15:38.833341 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:38.833384 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260925 15:15:38.833904 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:38.833682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:38243)
0925 15:15:38.833831 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:38.833861 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":45,"threads_started":1}
I20260925 15:15:39.834417 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:39.834129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:55191)
0925 15:15:39.834309 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:39.834357 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260925 15:15:39.834888 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:39.834121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:58097)
0925 15:15:39.834800 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:39.834838 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":76,"threads_started":1}
I20260925 15:15:39.834968 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:39.834415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:40227)
0925 15:15:39.834884 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:39.834922 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":44,"threads_started":1}
I20260925 15:15:40.818778 15775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 743e6dbca0ce4f3da82a3d4d3371b902)
I20260925 15:15:40.818873 15775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:40.819043 15775 leader_election.cc:288] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445)
I20260925 15:15:40.819468 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:40.819196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:34068)
0925 15:15:40.819376 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:40.819422 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260925 15:15:40.819900 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:40.819796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:34080)
0925 15:15:40.819846 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:40.819876 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:15:40.819981 15044 leader_election.cc:334] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445): Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:40.820070 15044 leader_election.cc:302] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 863d08373b7a4f6bb257e32b07b66d89; no voters: 743e6dbca0ce4f3da82a3d4d3371b902
I20260925 15:15:40.820218 15775 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:15:40.835793 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:40.835269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:38705)
0925 15:15:40.835685 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:40.835734 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":78,"threads_started":1}
I20260925 15:15:40.835904 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:40.835685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:54789)
0925 15:15:40.835783 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:40.835817 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260925 15:15:40.836057 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:40.835412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:47913)
0925 15:15:40.835977 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:40.836014 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":114,"threads_started":1}
I20260925 15:15:41.836759 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:41.836422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:56185)
0925 15:15:41.836666 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:41.836707 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":62,"threads_started":1}
I20260925 15:15:41.836759 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:41.836427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:44031)
0925 15:15:41.836667 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:41.836707 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":60,"threads_started":1}
I20260925 15:15:41.836763 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:41.836498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:47373)
0925 15:15:41.836673 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:41.836713 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":41,"threads_started":1}
I20260925 15:15:42.837517 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:42.837216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:38041)
0925 15:15:42.837413 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:42.837454 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260925 15:15:42.837710 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:42.837396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:51427)
0925 15:15:42.837502 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:42.837587 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":45,"threads_started":1}
I20260925 15:15:42.837939 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:42.837717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:36031)
0925 15:15:42.837857 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:42.837894 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":45,"threads_started":1}
I20260925 15:15:43.838335 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:43.838035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:50783)
0925 15:15:43.838223 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:43.838270 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260925 15:15:43.838622 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:43.838386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:44987)
0925 15:15:43.838495 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:43.838529 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260925 15:15:43.838995 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:43.838766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:51587)
0925 15:15:43.838915 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:43.838963 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260925 15:15:44.027060 15790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 743e6dbca0ce4f3da82a3d4d3371b902)
I20260925 15:15:44.027163 15790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:44.027401 15790 leader_election.cc:288] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445)
I20260925 15:15:44.027783 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:44.027592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:42964)
0925 15:15:44.027703 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:44.027747 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
W20260925 15:15:44.027864 15044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260925 15:15:44.028235 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:44.028120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:42972)
0925 15:15:44.028183 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:44.028217 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:15:44.028316 15044 leader_election.cc:334] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445): Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:44.028357 15044 leader_election.cc:302] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 863d08373b7a4f6bb257e32b07b66d89; no voters: 743e6dbca0ce4f3da82a3d4d3371b902
I20260925 15:15:44.028429 15790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:15:44.839260 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:44.839004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:57361)
0925 15:15:44.839139 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:44.839178 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260925 15:15:44.839324 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:44.839019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:37097)
0925 15:15:44.839210 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:44.839269 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":53,"threads_started":1}
I20260925 15:15:44.839709 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:44.839543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:57559)
0925 15:15:44.839634 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:44.839660 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":45,"threads_started":1}
I20260925 15:15:45.840101 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:45.839763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:60319)
0925 15:15:45.839985 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:45.840038 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":65,"threads_started":1}
I20260925 15:15:45.840353 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:45.840150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:54701)
0925 15:15:45.840283 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:45.840312 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260925 15:15:45.840350 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:45.840069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:55155)
0925 15:15:45.840255 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:45.840294 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":48,"threads_started":1}
I20260925 15:15:46.808101 15799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 743e6dbca0ce4f3da82a3d4d3371b902)
I20260925 15:15:46.808197 15799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:46.808308 15799 leader_election.cc:288] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445)
I20260925 15:15:46.808722 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:46.808487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:43104)
0925 15:15:46.808638 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:46.808680 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260925 15:15:46.809142 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:46.809045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:43110)
0925 15:15:46.809095 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:46.809123 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:15:46.809237 15044 leader_election.cc:334] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445): Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:46.809306 15044 leader_election.cc:302] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 863d08373b7a4f6bb257e32b07b66d89; no voters: 743e6dbca0ce4f3da82a3d4d3371b902
I20260925 15:15:46.809404 15799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:15:46.841102 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:46.840805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:40843)
0925 15:15:46.841002 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:46.841047 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
I20260925 15:15:46.841104 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:46.840731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:59271)
0925 15:15:46.841016 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:46.841057 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":85,"threads_started":1}
I20260925 15:15:46.841444 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:46.840946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:54559)
0925 15:15:46.841366 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:46.841400 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":43,"threads_started":1}
I20260925 15:15:47.842059 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:47.841680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:34117)
0925 15:15:47.841955 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:47.841999 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":64,"threads_started":1}
I20260925 15:15:47.842090 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:47.841737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:36299)
0925 15:15:47.842001 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:47.842038 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":56,"threads_started":1}
I20260925 15:15:47.842159 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:47.841838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:34283)
0925 15:15:47.842086 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:47.842120 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":43,"threads_started":1}
I20260925 15:15:48.842859 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:48.842650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:55943)
0925 15:15:48.842771 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:48.842804 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
I20260925 15:15:48.842876 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:48.842544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:55477)
0925 15:15:48.842812 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:48.842838 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260925 15:15:48.842885 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:48.842593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:44697)
0925 15:15:48.842749 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:48.842791 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260925 15:15:49.843520 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:49.843288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:50221)
0925 15:15:49.843427 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:49.843467 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260925 15:15:49.843710 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:49.843376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:52189)
0925 15:15:49.843626 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:49.843664 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":84,"threads_started":1}
I20260925 15:15:49.843798 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:49.843288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:56269)
0925 15:15:49.843722 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:49.843757 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":65,"threads_started":1}
I20260925 15:15:50.844251 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:50.843970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:42945)
0925 15:15:50.844122 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:50.844187 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260925 15:15:50.844368 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:50.844160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:33073)
0925 15:15:50.844286 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:50.844318 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260925 15:15:50.844439 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:50.844253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:49277)
0925 15:15:50.844371 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:50.844402 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
I20260925 15:15:51.465587 15817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 743e6dbca0ce4f3da82a3d4d3371b902)
I20260925 15:15:51.465677 15817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:51.465812 15817 leader_election.cc:288] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445)
I20260925 15:15:51.466162 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:51.465968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:43364)
0925 15:15:51.466074 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:51.466124 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":47,"threads_started":1}
W20260925 15:15:51.466276 15044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260925 15:15:51.466634 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:51.466555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:43368)
0925 15:15:51.466595 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:51.466619 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 15:15:51.466728 15044 leader_election.cc:334] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445): Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:51.466780 15044 leader_election.cc:302] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 863d08373b7a4f6bb257e32b07b66d89; no voters: 743e6dbca0ce4f3da82a3d4d3371b902
I20260925 15:15:51.466856 15817 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:15:51.844987 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:51.844739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:50047)
0925 15:15:51.844885 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:51.844927 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260925 15:15:51.845044 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:51.844739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:55253)
0925 15:15:51.844966 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:51.844996 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":70,"threads_started":1}
I20260925 15:15:51.845084 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:51.844846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:49751)
0925 15:15:51.844976 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:51.845017 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260925 15:15:52.845623 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.845410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:53941)
0925 15:15:52.845534 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:52.845570 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260925 15:15:52.845654 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.845410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:35657)
0925 15:15:52.845534 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:52.845570 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 15:15:52.845736 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.845472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:38987)
0925 15:15:52.845646 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:52.845682 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":51,"threads_started":1}
I20260925 15:15:53.846328 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:53.846106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:36809)
0925 15:15:53.846250 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:53.846283 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260925 15:15:53.846455 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:53.846274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:53525)
0925 15:15:53.846371 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:53.846408 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":57,"threads_started":1}
I20260925 15:15:53.846464 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:53.846069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:40135)
0925 15:15:53.846399 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:53.846428 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":66,"threads_started":1}
I20260925 15:15:54.846966 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:54.846713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:41671)
0925 15:15:54.846847 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:54.846909 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 15:15:54.847101 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:54.846854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:49833)
0925 15:15:54.847005 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:54.847047 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260925 15:15:54.847167 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:54.846912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:45239)
0925 15:15:54.847041 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:54.847125 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260925 15:15:54.873957 15832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 743e6dbca0ce4f3da82a3d4d3371b902)
I20260925 15:15:54.874043 15832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:54.874192 15832 leader_election.cc:288] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445)
I20260925 15:15:54.874564 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:54.874385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:37362)
0925 15:15:54.874486 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:54.874528 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
I20260925 15:15:54.874992 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:54.874922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:37372)
0925 15:15:54.874958 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:54.874980 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 15:15:54.875062 15044 leader_election.cc:334] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445): Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:54.875133 15044 leader_election.cc:302] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 863d08373b7a4f6bb257e32b07b66d89; no voters: 743e6dbca0ce4f3da82a3d4d3371b902
I20260925 15:15:54.875247 15832 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:15:55.847796 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.847537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:49615)
0925 15:15:55.847723 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:55.847752 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":54,"threads_started":1}
I20260925 15:15:55.847796 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.847589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:56565)
0925 15:15:55.847712 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:55.847752 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":46,"threads_started":1}
I20260925 15:15:55.847792 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.847411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:37979)
0925 15:15:55.847704 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:55.847750 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":64,"threads_started":1}
I20260925 15:15:56.848467 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.848176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:56175)
0925 15:15:56.848366 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:56.848409 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
I20260925 15:15:56.848490 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.848194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:40523)
0925 15:15:56.848398 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:56.848439 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":47,"threads_started":1}
I20260925 15:15:56.848605 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.848402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:49525)
0925 15:15:56.848537 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:56.848567 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":47,"threads_started":1}
I20260925 15:15:57.849181 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:57.848949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:55403)
0925 15:15:57.849044 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:57.849110 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":39,"threads_started":1}
I20260925 15:15:57.849192 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:57.848969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:35741)
0925 15:15:57.849125 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:57.849154 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":55,"threads_started":1}
I20260925 15:15:57.849171 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:57.848865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:57037)
0925 15:15:57.849003 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:57.849053 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260925 15:15:58.849861 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:58.849538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:34295)
0925 15:15:58.849767 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:58.849807 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":60,"threads_started":1}
I20260925 15:15:58.849932 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:58.849674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:32925)
0925 15:15:58.849812 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:58.849876 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260925 15:15:58.850149 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:58.849676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:36887)
0925 15:15:58.849948 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:58.850101 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":44,"threads_started":1}
I20260925 15:15:59.850729 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.850476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:57009)
0925 15:15:59.850627 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.850670 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260925 15:15:59.850790 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.850514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:33787)
0925 15:15:59.850709 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.850745 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
I20260925 15:15:59.850793 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.850514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:59555)
0925 15:15:59.850701 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.850738 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":53,"threads_started":1}
I20260925 15:15:59.935940 15850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 743e6dbca0ce4f3da82a3d4d3371b902)
I20260925 15:15:59.936033 15850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "743e6dbca0ce4f3da82a3d4d3371b902" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 34445 } } peers { permanent_uuid: "863d08373b7a4f6bb257e32b07b66d89" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42655 } attrs { promote: false } } peers { permanent_uuid: "9720f9afeca241b6a20e6489603d4d25" member_type: NON_VOTER last_known_addr { host: "127.8.43.188" port: 39137 } attrs { promote: true } }
I20260925 15:15:59.936226 15850 leader_election.cc:288] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445)
I20260925 15:15:59.936545 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.936366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:37586)
0925 15:15:59.936464 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.936507 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":45,"threads_started":1}
W20260925 15:15:59.936637 15044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260925 15:15:59.936995 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.936916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.0.0.1:37598)
0925 15:15:59.936948 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.936975 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 15:15:59.937071 15044 leader_election.cc:334] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 743e6dbca0ce4f3da82a3d4d3371b902 (127.8.43.190:34445): Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
I20260925 15:15:59.937110 15044 leader_election.cc:302] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 863d08373b7a4f6bb257e32b07b66d89; no voters: 743e6dbca0ce4f3da82a3d4d3371b902
I20260925 15:15:59.937202 15850 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 863d08373b7a4f6bb257e32b07b66d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:16:00.851495 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.851282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:39987)
0925 15:16:00.851406 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.851442 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260925 15:16:00.851495 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.851317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:38283)
0925 15:16:00.851406 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.851443 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":38,"threads_started":1}
I20260925 15:16:00.851688 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.851276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:59045)
0925 15:16:00.851554 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.851640 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":64,"threads_started":1}
I20260925 15:16:01.852155 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:01.851921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:45571)
0925 15:16:01.852052 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:01.852094 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260925 15:16:01.852275 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:01.851999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:48245)
0925 15:16:01.852199 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:01.852232 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":51,"threads_started":1}
I20260925 15:16:01.852406 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:01.852155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:44579)
0925 15:16:01.852313 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:01.852350 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":44,"threads_started":1}
I20260925 15:16:02.853068 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:02.852692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:51129)
0925 15:16:02.852964 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:02.853005 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":81,"threads_started":1}
I20260925 15:16:02.853065 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:02.852728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:58431)
0925 15:16:02.852966 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:02.853008 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":58,"threads_started":1}
I20260925 15:16:02.853170 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:02.852826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:47079)
0925 15:16:02.853066 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:02.853126 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":44,"threads_started":1}
I20260925 15:16:03.853876 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:03.853535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:41815)
0925 15:16:03.853737 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:03.853795 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260925 15:16:03.853924 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:03.853683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:49479)
0925 15:16:03.853848 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:03.853880 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":44,"threads_started":1}
I20260925 15:16:03.853906 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:03.853629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:39571)
0925 15:16:03.853814 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:03.853856 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":60,"threads_started":1}
I20260925 15:16:04.854574 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:04.854263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.131:57689)
0925 15:16:04.854411 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:04.854452 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260925 15:16:04.854638 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:04.854430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.129:45245)
0925 15:16:04.854539 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:04.854595 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":55,"threads_started":1}
I20260925 15:16:04.854638 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:04.854435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:34445 (local address 127.8.43.130:44055)
0925 15:16:04.854572 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:04.854598 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:34445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.8.43.190:34445,127.8.43.189:42655,127.8.43.188:39137, num_attempts: 1)
I20260925 15:16:05.027869 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 15204
I20260925 15:16:05.032897 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 15431
I20260925 15:16:05.037600 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 15567
I20260925 15:16:05.043108 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 14923
I20260925 15:16:05.047533 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 15111
2026-09-25T15:16:05Z chronyd exiting
I20260925 15:16:05.063283 8366 test_util.cc:175] -----------------------------------------------
I20260925 15:16:05.063346 8366 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349306597827-8366-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33070 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:16:05 dist-test-slave-c5d6 krb5kdc[15870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:16:05 dist-test-slave-c5d6 krb5kdc[15870](info): set up 2 sockets
Sep 25 15:16:05 dist-test-slave-c5d6 krb5kdc[15870](info): commencing operation
krb5kdc: starting...
W20260925 15:16:06.099709 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.190.keytab.
Entry for principal oryx/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.190.keytab.
I20260925 15:16:06.214155 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:32955
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.190.keytab
--principal=oryx/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:06.294438 15883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:06.294632 15883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:06.294653 15883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:06.295753 15883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:06.295786 15883 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:16:06.295799 15883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:06.295812 15883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:06.295825 15883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:06.295836 15883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:06.297281 15883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:32955
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.190.keytab
--principal=oryx/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:06.297484 15883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:06.297652 15883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:06.302932 15889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.302966 15888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.302966 15883 server_base.cc:1034] running on GCE node
W20260925 15:16:06.302984 15891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.303351 15883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:06.303699 15883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:06.304857 15883 hybrid_clock.cc:648] HybridClock initialized: now 1790349366304846 us; error 26 us; skew 500 ppm
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:06.307389 15883 init.cc:377] Logged in from keytab as oryx/127.8.43.190@KRBTEST.COM (short username oryx)
I20260925 15:16:06.307873 15883 webserver.cc:460] Webserver started at http://127.8.43.190:38405/ using document root <none> and password file <none>
I20260925 15:16:06.308117 15883 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:06.308195 15883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:06.308315 15883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:06.309298 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "34c120ebf3f347daaccdf201f2811801"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "558ea840ebc1a651ff126e1ea0877dce"
server_key_iv: "f14e768676b892b53c8d151e86356e4e"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.309587 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "34c120ebf3f347daaccdf201f2811801"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "558ea840ebc1a651ff126e1ea0877dce"
server_key_iv: "f14e768676b892b53c8d151e86356e4e"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.310809 15883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:16:06.311478 15898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.311699 15883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:06.311798 15883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "34c120ebf3f347daaccdf201f2811801"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "558ea840ebc1a651ff126e1ea0877dce"
server_key_iv: "f14e768676b892b53c8d151e86356e4e"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.311870 15883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:06.321274 15883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:06.323562 15883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:06.323704 15883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:06.328583 15883 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:32955
I20260925 15:16:06.328629 15960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:32955 every 8 connection(s)
I20260925 15:16:06.328965 15883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:06.329582 15961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:06.331763 15961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: Bootstrap starting.
I20260925 15:16:06.332340 15961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:06.332630 15961 log.cc:824] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:06.333287 15961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: No bootstrap required, opened a new log
I20260925 15:16:06.335935 15961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c120ebf3f347daaccdf201f2811801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 32955 } }
I20260925 15:16:06.336068 15961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:06.336091 15961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c120ebf3f347daaccdf201f2811801, State: Initialized, Role: FOLLOWER
I20260925 15:16:06.336234 15961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c120ebf3f347daaccdf201f2811801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 32955 } }
I20260925 15:16:06.336308 15961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:06.336333 15961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:06.336369 15961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:06.336949 15961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c120ebf3f347daaccdf201f2811801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 32955 } }
I20260925 15:16:06.337033 15961 leader_election.cc:302] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34c120ebf3f347daaccdf201f2811801; no voters:
I20260925 15:16:06.337213 15961 leader_election.cc:288] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:16:06.337260 15964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:06.337432 15964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [term 1 LEADER]: Becoming Leader. State: Replica: 34c120ebf3f347daaccdf201f2811801, State: Running, Role: LEADER
I20260925 15:16:06.337473 15961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:06.337550 15964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c120ebf3f347daaccdf201f2811801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 32955 } }
I20260925 15:16:06.337942 15965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34c120ebf3f347daaccdf201f2811801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c120ebf3f347daaccdf201f2811801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 32955 } } }
I20260925 15:16:06.338092 15965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:06.337952 15966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34c120ebf3f347daaccdf201f2811801. Latest consensus state: current_term: 1 leader_uuid: "34c120ebf3f347daaccdf201f2811801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c120ebf3f347daaccdf201f2811801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 32955 } } }
I20260925 15:16:06.338374 15966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:06.338429 15973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:06.338701 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 15883
I20260925 15:16:06.338817 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:16:06.338989 15973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:06.339012 8366 external_mini_cluster.cc:1428] Setting key 7fa4826ac1eb8c7bd53844348aad57e4
I20260925 15:16:06.340575 15973 catalog_manager.cc:1347] Generated new cluster ID: cc0bf2014a25452598736aa2002cb333
I20260925 15:16:06.340627 15973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:16:06.347173 15973 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:06.347919 15973 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:06.353528 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.339479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.8.43.190:32955)
0925 15:16:06.339637 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.339647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.339682 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:16:06.339848 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.339854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.339897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.340026 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.340747 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.342118 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.342844 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.343455 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.343556 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.346331 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.346335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.346335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.346347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.351994 (+ 5647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.352391 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.352394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.352394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.352490 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.352725 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.352726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.352726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.352943 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.353120 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.353209 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.353361 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260925 15:16:06.364645 15973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: Generated new TSK 0
I20260925 15:16:06.364854 15973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab.
Entry for principal oryx/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab.
I20260925 15:16:06.390020 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:32955
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab
--principal=oryx/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:06.470372 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:06.470592 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:06.470628 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:06.471828 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:06.471887 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:06.471920 15987 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:16:06.471959 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:16:06.473423 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab
--principal=oryx/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:06.473680 15987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:06.473905 15987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:06.478999 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.479029 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.479012 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.479043 15987 server_base.cc:1034] running on GCE node
I20260925 15:16:06.479444 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:06.479691 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:06.480865 15987 hybrid_clock.cc:648] HybridClock initialized: now 1790349366480848 us; error 37 us; skew 500 ppm
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:06.483446 15987 init.cc:377] Logged in from keytab as oryx/127.8.43.129@KRBTEST.COM (short username oryx)
I20260925 15:16:06.483946 15987 webserver.cc:460] Webserver started at http://127.8.43.129:39541/ using document root <none> and password file <none>
I20260925 15:16:06.484237 15987 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:06.484330 15987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:06.484464 15987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:06.485819 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "7d6b8c0d1b040a4fa88414a40b56dcca"
server_key_iv: "1f7a3061fdee97e5aa11cd922954651b"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.486179 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "7d6b8c0d1b040a4fa88414a40b56dcca"
server_key_iv: "1f7a3061fdee97e5aa11cd922954651b"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.487834 15987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 15:16:06.488575 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.488792 15987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:16:06.488864 15987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "7d6b8c0d1b040a4fa88414a40b56dcca"
server_key_iv: "1f7a3061fdee97e5aa11cd922954651b"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.488955 15987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:06.526223 15987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:06.528461 15987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:06.528580 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:06.529062 15987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:06.529101 15987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.529124 15987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:06.529140 15987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.535781 15987 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:37289
I20260925 15:16:06.535871 16117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:37289 every 8 connection(s)
I20260925 15:16:06.536191 15987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:16:06.539776 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.529704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:51813 (local address 127.8.43.190:32955)
0925 15:16:06.529763 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.529765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.529988 (+ 223us) server_negotiation.cc:407] Connection header received
0925 15:16:06.530641 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.530643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.530661 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.530734 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.531599 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.532482 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.534070 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.534438 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.534520 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.537145 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.537147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.537147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.537154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.538538 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.538909 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.538910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.538910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.539002 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.539285 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.539286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.539286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.539386 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.539439 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.539647 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.539710 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:16:06.540028 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.529698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:32955 (local address 127.8.43.129:51813)
0925 15:16:06.529865 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.529900 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.530492 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.530819 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.530823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.531195 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:06.531507 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.531512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.532581 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.532582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.533977 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.533979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.534528 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.534530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.534752 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.535358 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:06.535367 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:06.537063 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:06.538595 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.538600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.538602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.538844 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.539085 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.539086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.539086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.539192 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.539442 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:06.539447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:06.539554 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.539628 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.539775 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260925 15:16:06.540547 16118 heartbeater.cc:347] Connected to a master server at 127.8.43.190:32955
I20260925 15:16:06.540632 16118 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:06.540853 16118 heartbeater.cc:511] Master 127.8.43.190:32955 requested a full tablet report, sending...
I20260925 15:16:06.541361 15915 ts_manager.cc:194] Registered new tserver with Master: 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289)
I20260925 15:16:06.542217 15915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:51813
I20260925 15:16:06.545297 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 15987
I20260925 15:16:06.545406 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:16:06.545629 8366 external_mini_cluster.cc:1428] Setting key 5741a627312e206582ae3e8e217cf6e0
I20260925 15:16:06.546867 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.541526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:37493 (local address 127.8.43.190:32955)
0925 15:16:06.541580 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.541583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.541680 (+ 97us) server_negotiation.cc:407] Connection header received
0925 15:16:06.541708 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.541708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.541718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.541780 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.542037 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.542664 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.543453 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.543645 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.543729 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.544459 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.544460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.544461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.544467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.545770 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.546092 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.546093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.546093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.546142 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.546349 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.546350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.546350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.546450 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.546501 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.546720 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.546795 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:16:06.546897 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.541483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:32955 (local address 127.8.43.129:37493)
0925 15:16:06.541605 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.541621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.541673 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.541788 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.541789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.541932 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:06.541979 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.541980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.542767 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.542767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.543373 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.543375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.543732 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.543733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.543794 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.544112 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:06.544115 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:06.544375 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:06.545833 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.545834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.545835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.546005 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.546217 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.546218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.546218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.546277 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.546516 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:06.546516 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:06.546561 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.546631 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.546737 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab.
Entry for principal oryx/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab.
I20260925 15:16:06.570091 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:32955
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab
--principal=oryx/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:06.653623 16127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:06.653833 16127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:06.653863 16127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:06.655035 16127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:06.655077 16127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:06.655097 16127 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:16:06.655114 16127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:16:06.656526 16127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab
--principal=oryx/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:06.656723 16127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:06.656893 16127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:06.661782 16132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.661789 16133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.661893 16135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.661968 16127 server_base.cc:1034] running on GCE node
I20260925 15:16:06.662236 16127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:06.662432 16127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:06.663597 16127 hybrid_clock.cc:648] HybridClock initialized: now 1790349366663575 us; error 32 us; skew 500 ppm
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:06.666581 16127 init.cc:377] Logged in from keytab as oryx/127.8.43.130@KRBTEST.COM (short username oryx)
I20260925 15:16:06.667013 16127 webserver.cc:460] Webserver started at http://127.8.43.130:41017/ using document root <none> and password file <none>
I20260925 15:16:06.667200 16127 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:06.667315 16127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:06.667424 16127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:06.668447 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "263c5fc128ac481bb56519c32889c3ac"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "aa31d1728f765067c8ad25dda21c194e"
server_key_iv: "2bccb89df25ad6c6c61e7cf4112afc38"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.668766 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "263c5fc128ac481bb56519c32889c3ac"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "aa31d1728f765067c8ad25dda21c194e"
server_key_iv: "2bccb89df25ad6c6c61e7cf4112afc38"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.670078 16127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:16:06.670810 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.671025 16127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:06.671137 16127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "263c5fc128ac481bb56519c32889c3ac"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "aa31d1728f765067c8ad25dda21c194e"
server_key_iv: "2bccb89df25ad6c6c61e7cf4112afc38"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.671249 16127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:06.698309 16127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:06.700816 16127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:06.700976 16127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:06.701682 16127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:06.701737 16127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.701773 16127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:06.701802 16127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:16:06.709010 16127 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:43667
I20260925 15:16:06.709087 16257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:43667 every 8 connection(s)
I20260925 15:16:06.709429 16127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:06.712291 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.702249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:36495 (local address 127.8.43.190:32955)
0925 15:16:06.702300 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.702303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.702501 (+ 198us) server_negotiation.cc:407] Connection header received
0925 15:16:06.702990 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.702991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.703005 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.703082 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.703804 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.704630 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.706239 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.706577 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.706649 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.709461 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.709462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.709462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.709470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.710872 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.711323 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.711324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.711325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.711386 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.711646 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.711647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.711647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.711781 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.711849 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.712126 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.712204 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:16:06.712487 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.702237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:32955 (local address 127.8.43.130:36495)
0925 15:16:06.702386 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.702420 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.702889 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.703167 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.703170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.703496 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:06.703681 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.703686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.704770 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.704772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.706165 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.706168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.706720 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.706721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.706935 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.707495 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:06.707513 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:06.709359 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:06.710972 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.710978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.710980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.711200 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.711454 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.711454 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.711455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.711564 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.711853 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:06.711859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:06.711990 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.712087 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.712245 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260925 15:16:06.712945 16258 heartbeater.cc:347] Connected to a master server at 127.8.43.190:32955
I20260925 15:16:06.713030 16258 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:06.713217 16258 heartbeater.cc:511] Master 127.8.43.190:32955 requested a full tablet report, sending...
I20260925 15:16:06.713587 15914 ts_manager.cc:194] Registered new tserver with Master: 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667)
I20260925 15:16:06.714217 15914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:36495
I20260925 15:16:06.715095 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 16127
I20260925 15:16:06.715212 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:16:06.715438 8366 external_mini_cluster.cc:1428] Setting key 801bfb58a55c7a4de2870ff788363364
I20260925 15:16:06.720122 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.713963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:58209 (local address 127.8.43.190:32955)
0925 15:16:06.714007 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.714009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.714066 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:16:06.714145 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.714146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.714173 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.714245 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.714583 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.715105 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.716054 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.716291 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.716343 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.717347 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.717348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.717349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.717354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.718951 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.719311 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.719312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.719313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.719363 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.719592 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.719593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.719593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.719738 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.719815 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.719966 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.720035 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:16:06.720136 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.713886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:32955 (local address 127.8.43.130:58209)
0925 15:16:06.713996 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.714005 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.714080 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.714245 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.714245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.714441 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:06.714502 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.714503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.715205 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.715206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.715954 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.715956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.716364 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.716366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.716441 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.716804 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:06.716827 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:06.717252 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:06.719029 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.719030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.719031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.719206 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.719461 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.719462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.719462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.719530 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.719808 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:06.719809 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:06.719854 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.719941 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.720037 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab.
Entry for principal oryx/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab.
I20260925 15:16:06.741680 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:32955
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab
--principal=oryx/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:06.821165 16267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:06.821352 16267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:06.821375 16267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:06.822423 16267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:06.822458 16267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:06.822477 16267 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:16:06.822495 16267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:16:06.823884 16267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab
--principal=oryx/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.43.190:32955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:06.824050 16267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:06.824218 16267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:06.828967 16272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.828980 16273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.829213 16267 server_base.cc:1034] running on GCE node
W20260925 15:16:06.828984 16275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.829528 16267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:06.829717 16267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:06.830852 16267 hybrid_clock.cc:648] HybridClock initialized: now 1790349366830832 us; error 38 us; skew 500 ppm
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:06.833184 16267 init.cc:377] Logged in from keytab as oryx/127.8.43.131@KRBTEST.COM (short username oryx)
I20260925 15:16:06.833595 16267 webserver.cc:460] Webserver started at http://127.8.43.131:35681/ using document root <none> and password file <none>
I20260925 15:16:06.833770 16267 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:06.833808 16267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:06.833894 16267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:06.834847 16267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "4972c60993054eccb391337aefdb127b"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "b391541f3b3d9bd294e45420ab058a11"
server_key_iv: "949c416bdfba6a6e649a0be01523207f"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.835132 16267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "4972c60993054eccb391337aefdb127b"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "b391541f3b3d9bd294e45420ab058a11"
server_key_iv: "949c416bdfba6a6e649a0be01523207f"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.836529 16267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:06.837241 16282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.837406 16267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:06.837488 16267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "4972c60993054eccb391337aefdb127b"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-c5d6"
server_key: "b391541f3b3d9bd294e45420ab058a11"
server_key_iv: "949c416bdfba6a6e649a0be01523207f"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.837541 16267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:06.856791 16267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:06.858502 16267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:06.858605 16267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:06.859018 16267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:06.859055 16267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.859076 16267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:06.859117 16267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.866259 16267 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:38787
I20260925 15:16:06.866324 16397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:38787 every 8 connection(s)
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:16:06.866633 16267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:06.869431 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.859472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:60885 (local address 127.8.43.190:32955)
0925 15:16:06.859542 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.859544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.859810 (+ 266us) server_negotiation.cc:407] Connection header received
0925 15:16:06.860534 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.860536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.860547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.860594 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.861530 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.862328 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.864019 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.864285 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.864343 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.867130 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.867132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.867132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.867139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.868335 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.868678 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.868679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.868679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.868755 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.868960 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.868961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.868961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.869061 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.869108 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.869313 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.869369 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:16:06.869655 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.859450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:32955 (local address 127.8.43.131:60885)
0925 15:16:06.859655 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.859713 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.860410 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.860699 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.860705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.861108 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:06.861409 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.861417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.862429 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.862431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.863927 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.863931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.864376 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.864377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.864549 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.865124 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:06.865134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:06.867036 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:06.868404 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.868409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.868411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.868613 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.868813 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.868814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.868814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.868907 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.869100 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:06.869103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:06.869221 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.869294 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.869405 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260925 15:16:06.870090 16398 heartbeater.cc:347] Connected to a master server at 127.8.43.190:32955
I20260925 15:16:06.870194 16398 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:06.870364 16398 heartbeater.cc:511] Master 127.8.43.190:32955 requested a full tablet report, sending...
I20260925 15:16:06.870723 15915 ts_manager.cc:194] Registered new tserver with Master: 4972c60993054eccb391337aefdb127b (127.8.43.131:38787)
I20260925 15:16:06.871133 15915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:60885
I20260925 15:16:06.876273 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 16267
I20260925 15:16:06.876379 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:16:06.876447 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.871071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:32955 (local address 127.8.43.131:59013)
0925 15:16:06.871175 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.871185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.871276 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.871404 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.871405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.871574 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:06.871624 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.871625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.872260 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.872260 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.872781 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.872782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.873159 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.873160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.873238 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.873643 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:06.873648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:06.873912 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:06.875441 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.875442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.875443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.875606 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.875820 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:06.875821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:06.875821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:06.875877 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:06.876140 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:06.876141 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:06.876179 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.876243 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.876317 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":60,"threads_started":1}
I20260925 15:16:06.876461 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.871128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:59013 (local address 127.8.43.190:32955)
0925 15:16:06.871166 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.871168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.871263 (+ 95us) server_negotiation.cc:407] Connection header received
0925 15:16:06.871334 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.871335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.871345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.871399 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.871714 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.872168 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.872875 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.873075 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.873156 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.874076 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.874078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.874078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.874084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.875342 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.875689 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.875697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.875697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.875748 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.875947 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.875948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.875948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.876055 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.876140 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.876288 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.876363 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:16:06.876573 8366 external_mini_cluster.cc:1428] Setting key 99bb7e351117b1f8bece7e0a812fa03b
I20260925 15:16:06.877427 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:16:06.898180 15983 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:06.892664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.8.43.190:32955)
0925 15:16:06.892734 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.892736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.892746 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:16:06.892781 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.892781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.892790 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.892842 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.893373 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.893898 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.894643 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.894829 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.894878 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.898062 (+ 3184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49854: BlockingRecv error: recv got EOF from 127.0.0.1:49854 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 15:16:06 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:16:06.905773 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.899019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.8.43.190:32955)
0925 15:16:06.899082 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.899084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.899199 (+ 115us) server_negotiation.cc:407] Connection header received
0925 15:16:06.899299 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.899300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.899310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.899360 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.899875 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.900343 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.901049 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.901236 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.901318 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.903499 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.903500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.903501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.903508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.904626 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.905021 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.905022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.905023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.905078 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.905281 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.905281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.905282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.905388 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.905488 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.905592 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.905662 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:16:06.907423 15915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:06.909006 15915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:06.915859 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.911424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:38787 (local address 127.0.0.1:44846)
0925 15:16:06.911525 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.911567 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.911627 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.912000 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.912003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.912006 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:06.912070 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.912072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.913389 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.913390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.914592 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.914594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.915519 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.915520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.915606 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.915620 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.915711 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.915765 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":53,"thread_start_us":46,"threads_started":1}
I20260925 15:16:06.915892 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.911541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44846 (local address 127.8.43.131:38787)
0925 15:16:06.911797 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.911801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.911809 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:16:06.911880 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.911883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.911899 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.911979 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:06.912155 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.913289 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.914678 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.915441 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.915516 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.915588 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.915725 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.915772 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260925 15:16:06.916088 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.911435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:43667 (local address 127.0.0.1:41702)
0925 15:16:06.912061 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.912071 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.912126 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.912357 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.912358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.912360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:06.912407 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.912409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.913330 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.913331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.914852 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.914853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.915842 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.915843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.915908 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.915911 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.915966 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.916012 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":85,"thread_start_us":86,"threads_started":1}
I20260925 15:16:06.916311 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.911620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:37289 (local address 127.0.0.1:52258)
0925 15:16:06.912210 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:06.912218 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:06.912256 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:06.912492 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:06.912494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:06.912496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:06.912532 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.912533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.913741 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.913742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.914847 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:06.914850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.915880 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.915881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:06.916114 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.916120 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:16:06.916181 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:16:06.916227 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260925 15:16:06.916332 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.911759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.8.43.130:43667)
0925 15:16:06.911939 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.911942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.912123 (+ 181us) server_negotiation.cc:407] Connection header received
0925 15:16:06.912259 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.912262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.912279 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.912359 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:06.912462 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.913238 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.914945 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.915684 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.915744 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.915809 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.916200 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.916238 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260925 15:16:06.916638 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.911731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.8.43.129:37289)
0925 15:16:06.911905 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.911909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.912306 (+ 397us) server_negotiation.cc:407] Connection header received
0925 15:16:06.912346 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.912348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.912363 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.912432 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:06.912598 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.913664 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.914932 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.915712 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.915775 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.915842 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.916515 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.916556 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260925 15:16:06.916713 16332 tablet_service.cc:1467] Processing CreateTablet for tablet 01c5437ec4054d5b8442cfd5a6ca08cb (DEFAULT_TABLE table=test-table [id=39bb2a2b1114461d80bc7aa971c1a9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:06.917078 16332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c5437ec4054d5b8442cfd5a6ca08cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:06.917228 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 01c5437ec4054d5b8442cfd5a6ca08cb (DEFAULT_TABLE table=test-table [id=39bb2a2b1114461d80bc7aa971c1a9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:06.917491 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c5437ec4054d5b8442cfd5a6ca08cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:06.919929 16192 tablet_service.cc:1467] Processing CreateTablet for tablet 01c5437ec4054d5b8442cfd5a6ca08cb (DEFAULT_TABLE table=test-table [id=39bb2a2b1114461d80bc7aa971c1a9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:06.920063 16426 tablet_bootstrap.cc:492] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Bootstrap starting.
I20260925 15:16:06.920235 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c5437ec4054d5b8442cfd5a6ca08cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:06.920600 16426 tablet_bootstrap.cc:654] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:06.920763 16427 tablet_bootstrap.cc:492] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Bootstrap starting.
I20260925 15:16:06.920941 16426 log.cc:824] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:06.921330 16427 tablet_bootstrap.cc:654] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:06.921617 16427 log.cc:824] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:06.921774 16426 tablet_bootstrap.cc:492] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: No bootstrap required, opened a new log
I20260925 15:16:06.921835 16426 ts_tablet_manager.cc:1397] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:16:06.922354 16427 tablet_bootstrap.cc:492] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: No bootstrap required, opened a new log
I20260925 15:16:06.922408 16427 ts_tablet_manager.cc:1397] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:16:06.923811 16430 tablet_bootstrap.cc:492] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Bootstrap starting.
I20260925 15:16:06.923854 16426 raft_consensus.cc:348] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:06.924021 16426 raft_consensus.cc:374] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:06.924093 16426 raft_consensus.cc:729] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4972c60993054eccb391337aefdb127b, State: Initialized, Role: FOLLOWER
I20260925 15:16:06.924228 16426 consensus_queue.cc:260] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:06.924454 16430 tablet_bootstrap.cc:654] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:06.924383 16427 raft_consensus.cc:348] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:06.924496 16426 ts_tablet_manager.cc:1428] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:06.924522 16427 raft_consensus.cc:374] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:06.924563 16427 raft_consensus.cc:729] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46720c1d83f54bd2a9fa0d9ccb38a02c, State: Initialized, Role: FOLLOWER
I20260925 15:16:06.924602 16398 heartbeater.cc:503] Master 127.8.43.190:32955 was elected leader, sending a full tablet report...
I20260925 15:16:06.924703 16427 consensus_queue.cc:260] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:06.924767 16430 log.cc:824] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:06.925027 16427 ts_tablet_manager.cc:1428] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:06.925048 16118 heartbeater.cc:503] Master 127.8.43.190:32955 was elected leader, sending a full tablet report...
I20260925 15:16:06.925704 16430 tablet_bootstrap.cc:492] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: No bootstrap required, opened a new log
I20260925 15:16:06.925774 16430 ts_tablet_manager.cc:1397] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:16:06.927485 16430 raft_consensus.cc:348] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:06.927599 16430 raft_consensus.cc:374] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:06.927620 16430 raft_consensus.cc:729] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263c5fc128ac481bb56519c32889c3ac, State: Initialized, Role: FOLLOWER
I20260925 15:16:06.927752 16430 consensus_queue.cc:260] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:06.928013 16430 ts_tablet_manager.cc:1428] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:16:06.928027 16258 heartbeater.cc:503] Master 127.8.43.190:32955 was elected leader, sending a full tablet report...
W20260925 15:16:06.960099 16259 tablet.cc:2369] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:07.036957 16119 tablet.cc:2369] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:07.117295 16399 tablet.cc:2369] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:07.151043 16431 raft_consensus.cc:482] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:07.151211 16431 raft_consensus.cc:504] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.151620 16431 leader_election.cc:288] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667), 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289)
I20260925 15:16:07.154572 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.151922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:33763 (local address 127.8.43.129:37289)
0925 15:16:07.152005 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.152007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.152018 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:16:07.152079 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.152080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.152100 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.152184 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:07.152355 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.152847 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.153827 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.154195 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.154286 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.154295 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.154462 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.154497 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:16:07.154570 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.151933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:53457 (local address 127.8.43.130:43667)
0925 15:16:07.151991 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.151994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.152195 (+ 201us) server_negotiation.cc:407] Connection header received
0925 15:16:07.152274 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.152275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.152286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.152355 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:07.152511 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.152975 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.153888 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.154244 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.154319 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.154327 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.154471 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.154505 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:16:07.154587 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.151888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:37289 (local address 127.8.43.131:33763)
0925 15:16:07.151939 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.151952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.152004 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.152201 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.152201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.152203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:07.152278 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.152280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.152943 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.152944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.153726 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.153728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.154283 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.154283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.154381 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.154385 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.154457 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.154498 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":22}
I20260925 15:16:07.154603 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.151888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:43667 (local address 127.8.43.131:53457)
0925 15:16:07.152092 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.152104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.152208 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.152358 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.152359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.152360 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:07.152439 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.152440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.153059 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.153060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.153790 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.153792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.154319 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.154319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.154397 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.154400 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.154465 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.154502 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":53,"thread_start_us":102,"threads_started":1}
I20260925 15:16:07.155009 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c5437ec4054d5b8442cfd5a6ca08cb" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" is_pre_election: true
I20260925 15:16:07.155194 16072 raft_consensus.cc:2454] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 0.
I20260925 15:16:07.155270 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c5437ec4054d5b8442cfd5a6ca08cb" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac" is_pre_election: true
I20260925 15:16:07.155441 16212 raft_consensus.cc:2454] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 0.
I20260925 15:16:07.155478 16285 leader_election.cc:302] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46720c1d83f54bd2a9fa0d9ccb38a02c, 4972c60993054eccb391337aefdb127b; no voters:
I20260925 15:16:07.155598 16431 raft_consensus.cc:2781] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:07.155669 16431 raft_consensus.cc:482] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:07.155699 16431 raft_consensus.cc:3037] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.156740 16431 raft_consensus.cc:504] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.156883 16431 leader_election.cc:288] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 election: Requested vote from peers 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667), 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289)
I20260925 15:16:07.157043 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c5437ec4054d5b8442cfd5a6ca08cb" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac"
I20260925 15:16:07.157141 16212 raft_consensus.cc:3037] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.157886 16212 raft_consensus.cc:2454] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 1.
I20260925 15:16:07.158018 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c5437ec4054d5b8442cfd5a6ca08cb" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c"
I20260925 15:16:07.158102 16283 leader_election.cc:302] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263c5fc128ac481bb56519c32889c3ac, 4972c60993054eccb391337aefdb127b; no voters:
I20260925 15:16:07.158109 16072 raft_consensus.cc:3037] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.158206 16431 raft_consensus.cc:2781] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:07.158340 16431 raft_consensus.cc:686] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 1 LEADER]: Becoming Leader. State: Replica: 4972c60993054eccb391337aefdb127b, State: Running, Role: LEADER
I20260925 15:16:07.158447 16431 consensus_queue.cc:237] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.159008 16072 raft_consensus.cc:2454] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 1.
I20260925 15:16:07.159328 15915 catalog_manager.cc:5495] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4972c60993054eccb391337aefdb127b (127.8.43.131). New cstate: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20260925 15:16:07.196907 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.194536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.8.43.131:38787)
0925 15:16:07.194596 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.194598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.194612 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:16:07.194660 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.194661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.194677 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.194767 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:07.194885 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.195368 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.196122 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.196322 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.196398 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.196564 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:07.196673 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:07.196771 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.196797 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.196815 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:16:07.198644 16072 raft_consensus.cc:1261] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Refusing update from remote peer 4972c60993054eccb391337aefdb127b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.198673 16212 raft_consensus.cc:1261] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Refusing update from remote peer 4972c60993054eccb391337aefdb127b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.199003 16436 consensus_queue.cc:1035] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [LEADER]: Connected to new peer: Peer: permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.199139 16431 consensus_queue.cc:1035] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [LEADER]: Connected to new peer: Peer: permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.201375 16439 mvcc.cc:205] Tried to move back new op lower bound from 7333271008042950656 to 7333271007881482240. Current Snapshot: MvccSnapshot[applied={T|T < 7333271008042950656}]
I20260925 15:16:07.202159 16440 mvcc.cc:205] Tried to move back new op lower bound from 7333271008042950656 to 7333271007881482240. Current Snapshot: MvccSnapshot[applied={T|T < 7333271008042950656 or (T in {7333271008042950656})}]
I20260925 15:16:07.203797 15914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:07.203897 15914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:07.204664 15914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: Sending DeleteTablet for 3 replicas of tablet 01c5437ec4054d5b8442cfd5a6ca08cb
I20260925 15:16:07.205009 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c5437ec4054d5b8442cfd5a6ca08cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:41702
I20260925 15:16:07.205089 16332 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c5437ec4054d5b8442cfd5a6ca08cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:44846
I20260925 15:16:07.205320 16450 tablet_replica.cc:330] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: stopping tablet replica
I20260925 15:16:07.205324 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c5437ec4054d5b8442cfd5a6ca08cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:52258
I20260925 15:16:07.205305 15914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:07.205442 15914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:07.205452 16450 raft_consensus.cc:2229] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:07.205559 16450 raft_consensus.cc:2258] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.206056 16450 ts_tablet_manager.cc:1905] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.206259 16449 tablet_replica.cc:330] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: stopping tablet replica
I20260925 15:16:07.206396 16449 raft_consensus.cc:2229] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:07.206530 16449 raft_consensus.cc:2258] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.206876 16192 tablet_service.cc:1467] Processing CreateTablet for tablet ff07b6ddd973454499dc2443edd3046a (DEFAULT_TABLE table=test-table [id=ba409b963ab84ac897b24b6a4a0b8ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.206997 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff07b6ddd973454499dc2443edd3046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.207041 16449 ts_tablet_manager.cc:1905] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.207563 16052 tablet_service.cc:1467] Processing CreateTablet for tablet ff07b6ddd973454499dc2443edd3046a (DEFAULT_TABLE table=test-table [id=ba409b963ab84ac897b24b6a4a0b8ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.207669 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff07b6ddd973454499dc2443edd3046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.207930 16450 ts_tablet_manager.cc:1918] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:07.207983 16450 log.cc:1196] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/01c5437ec4054d5b8442cfd5a6ca08cb
I20260925 15:16:07.208220 16450 ts_tablet_manager.cc:1939] T 01c5437ec4054d5b8442cfd5a6ca08cb P 4972c60993054eccb391337aefdb127b: Deleting consensus metadata
I20260925 15:16:07.208613 16449 ts_tablet_manager.cc:1918] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:07.208683 16449 log.cc:1196] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/01c5437ec4054d5b8442cfd5a6ca08cb
I20260925 15:16:07.208809 15901 catalog_manager.cc:4841] TS 4972c60993054eccb391337aefdb127b (127.8.43.131:38787): tablet 01c5437ec4054d5b8442cfd5a6ca08cb (table test-table [id=39bb2a2b1114461d80bc7aa971c1a9c4]) successfully deleted
I20260925 15:16:07.208931 16449 ts_tablet_manager.cc:1939] T 01c5437ec4054d5b8442cfd5a6ca08cb P 263c5fc128ac481bb56519c32889c3ac: Deleting consensus metadata
I20260925 15:16:07.209095 16427 tablet_bootstrap.cc:492] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: Bootstrap starting.
I20260925 15:16:07.209280 15899 catalog_manager.cc:4841] TS 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667): tablet 01c5437ec4054d5b8442cfd5a6ca08cb (table test-table [id=39bb2a2b1114461d80bc7aa971c1a9c4]) successfully deleted
I20260925 15:16:07.209456 16427 tablet_bootstrap.cc:654] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.209446 16332 tablet_service.cc:1467] Processing CreateTablet for tablet ff07b6ddd973454499dc2443edd3046a (DEFAULT_TABLE table=test-table [id=ba409b963ab84ac897b24b6a4a0b8ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.209538 16332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff07b6ddd973454499dc2443edd3046a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.210160 16427 tablet_bootstrap.cc:492] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: No bootstrap required, opened a new log
I20260925 15:16:07.210215 16427 ts_tablet_manager.cc:1397] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:07.210361 16427 raft_consensus.cc:348] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.210428 16427 raft_consensus.cc:374] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.210456 16427 raft_consensus.cc:729] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46720c1d83f54bd2a9fa0d9ccb38a02c, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.210495 16427 consensus_queue.cc:260] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.210608 16427 ts_tablet_manager.cc:1428] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.210731 16426 tablet_bootstrap.cc:492] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: Bootstrap starting.
I20260925 15:16:07.211122 16426 tablet_bootstrap.cc:654] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.211292 16451 tablet_replica.cc:330] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: stopping tablet replica
I20260925 15:16:07.211383 16451 raft_consensus.cc:2229] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:07.211468 16451 raft_consensus.cc:2258] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.211922 16451 ts_tablet_manager.cc:1905] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.211946 16426 tablet_bootstrap.cc:492] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: No bootstrap required, opened a new log
I20260925 15:16:07.211961 16430 tablet_bootstrap.cc:492] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: Bootstrap starting.
I20260925 15:16:07.211989 16426 ts_tablet_manager.cc:1397] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:07.212144 16426 raft_consensus.cc:348] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.212221 16426 raft_consensus.cc:374] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.212244 16426 raft_consensus.cc:729] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4972c60993054eccb391337aefdb127b, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.212283 16426 consensus_queue.cc:260] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.212370 16430 tablet_bootstrap.cc:654] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.212378 16426 ts_tablet_manager.cc:1428] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.213070 16430 tablet_bootstrap.cc:492] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: No bootstrap required, opened a new log
I20260925 15:16:07.213133 16430 ts_tablet_manager.cc:1397] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:07.213269 16430 raft_consensus.cc:348] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.213313 16430 raft_consensus.cc:374] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.213325 16430 raft_consensus.cc:729] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263c5fc128ac481bb56519c32889c3ac, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.213379 16430 consensus_queue.cc:260] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.213459 16430 ts_tablet_manager.cc:1428] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.213739 16451 ts_tablet_manager.cc:1918] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:07.213840 16451 log.cc:1196] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/01c5437ec4054d5b8442cfd5a6ca08cb
I20260925 15:16:07.214509 16451 ts_tablet_manager.cc:1939] T 01c5437ec4054d5b8442cfd5a6ca08cb P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting consensus metadata
I20260925 15:16:07.214882 15901 catalog_manager.cc:4841] TS 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289): tablet 01c5437ec4054d5b8442cfd5a6ca08cb (table test-table [id=39bb2a2b1114461d80bc7aa971c1a9c4]) successfully deleted
I20260925 15:16:07.308022 16444 raft_consensus.cc:482] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:07.308145 16444 raft_consensus.cc:504] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.308360 16444 leader_election.cc:288] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289), 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667)
I20260925 15:16:07.308563 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff07b6ddd973454499dc2443edd3046a" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac" is_pre_election: true
I20260925 15:16:07.308557 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff07b6ddd973454499dc2443edd3046a" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" is_pre_election: true
I20260925 15:16:07.308653 16212 raft_consensus.cc:2454] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 0.
I20260925 15:16:07.308653 16072 raft_consensus.cc:2454] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 0.
I20260925 15:16:07.308832 16283 leader_election.cc:302] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263c5fc128ac481bb56519c32889c3ac, 4972c60993054eccb391337aefdb127b; no voters:
I20260925 15:16:07.308960 16444 raft_consensus.cc:2781] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:07.309010 16444 raft_consensus.cc:482] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:07.309041 16444 raft_consensus.cc:3037] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.309711 16444 raft_consensus.cc:504] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.309849 16444 leader_election.cc:288] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 election: Requested vote from peers 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289), 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667)
I20260925 15:16:07.309971 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff07b6ddd973454499dc2443edd3046a" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c"
I20260925 15:16:07.310070 16072 raft_consensus.cc:3037] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.310050 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff07b6ddd973454499dc2443edd3046a" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac"
I20260925 15:16:07.310145 16212 raft_consensus.cc:3037] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.310676 16072 raft_consensus.cc:2454] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 1.
I20260925 15:16:07.310833 16285 leader_election.cc:302] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46720c1d83f54bd2a9fa0d9ccb38a02c, 4972c60993054eccb391337aefdb127b; no voters:
I20260925 15:16:07.310933 16444 raft_consensus.cc:2781] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:07.310977 16444 raft_consensus.cc:686] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 1 LEADER]: Becoming Leader. State: Replica: 4972c60993054eccb391337aefdb127b, State: Running, Role: LEADER
I20260925 15:16:07.311040 16212 raft_consensus.cc:2454] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 1.
I20260925 15:16:07.311029 16444 consensus_queue.cc:237] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.311676 15914 catalog_manager.cc:5495] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4972c60993054eccb391337aefdb127b (127.8.43.131). New cstate: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } health_report { overall_health: HEALTHY } } }
Sep 25 15:16:07 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.190@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:16:07.362847 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.355992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:32955 (local address 127.0.0.1:49876)
0925 15:16:07.356230 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.356243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.356349 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.356518 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.356519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.356868 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:07.356919 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.356922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.357698 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.357699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.358218 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.358219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.358589 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.358590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.358648 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.359133 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:07.359138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:07.360531 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:07.361943 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:07.361949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:07.361951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:07.362142 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:07.362312 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:07.362312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:07.362313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:07.362366 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:07.362564 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:07.362570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:07.362608 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.362659 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.362747 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260925 15:16:07.362864 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.356090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.8.43.190:32955)
0925 15:16:07.356209 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.356211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.356328 (+ 117us) server_negotiation.cc:407] Connection header received
0925 15:16:07.356410 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.356411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.356424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.356501 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:07.357038 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.357544 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.358318 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.358501 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.358590 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.360613 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:07.360615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:07.360615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:07.360621 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:07.361871 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:07.362209 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:07.362210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:07.362210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:07.362256 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:07.362419 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:07.362419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:07.362419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:07.362513 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:07.362561 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.362712 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.362791 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":74}
I20260925 15:16:07.364938 15914 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:49876:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:16:07.365162 15914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:07.366649 16332 tablet_service.cc:1467] Processing CreateTablet for tablet a5a4a8b553844698992b30977b42958e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3045832f36994fa3b2a57f4dcad258f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:16:07.366675 16192 tablet_service.cc:1467] Processing CreateTablet for tablet a5a4a8b553844698992b30977b42958e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3045832f36994fa3b2a57f4dcad258f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:16:07.366740 16052 tablet_service.cc:1467] Processing CreateTablet for tablet a5a4a8b553844698992b30977b42958e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3045832f36994fa3b2a57f4dcad258f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:16:07.366828 16332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a4a8b553844698992b30977b42958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.366830 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a4a8b553844698992b30977b42958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.366875 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a4a8b553844698992b30977b42958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.368274 16426 tablet_bootstrap.cc:492] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: Bootstrap starting.
I20260925 15:16:07.368371 16430 tablet_bootstrap.cc:492] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: Bootstrap starting.
I20260925 15:16:07.368625 16426 tablet_bootstrap.cc:654] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.368719 16430 tablet_bootstrap.cc:654] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.369052 16427 tablet_bootstrap.cc:492] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Bootstrap starting.
I20260925 15:16:07.369225 16426 tablet_bootstrap.cc:492] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: No bootstrap required, opened a new log
I20260925 15:16:07.369277 16426 ts_tablet_manager.cc:1397] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:07.369385 16426 raft_consensus.cc:348] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.369467 16426 raft_consensus.cc:374] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.369464 16427 tablet_bootstrap.cc:654] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.369489 16426 raft_consensus.cc:729] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4972c60993054eccb391337aefdb127b, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.369549 16426 consensus_queue.cc:260] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.369680 16426 ts_tablet_manager.cc:1428] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:07.369820 16431 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.369899 16431 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:16:07.370397 16427 tablet_bootstrap.cc:492] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: No bootstrap required, opened a new log
I20260925 15:16:07.370451 16427 ts_tablet_manager.cc:1397] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:07.370601 16427 raft_consensus.cc:348] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.370687 16427 raft_consensus.cc:374] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.370709 16427 raft_consensus.cc:729] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46720c1d83f54bd2a9fa0d9ccb38a02c, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.370750 16427 consensus_queue.cc:260] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.370848 16427 ts_tablet_manager.cc:1428] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.370899 16432 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.370962 16432 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:16:07.371510 16430 tablet_bootstrap.cc:492] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: No bootstrap required, opened a new log
I20260925 15:16:07.371562 16430 ts_tablet_manager.cc:1397] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:16:07.371721 16430 raft_consensus.cc:348] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.371790 16430 raft_consensus.cc:374] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.371810 16430 raft_consensus.cc:729] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263c5fc128ac481bb56519c32889c3ac, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.371860 16430 consensus_queue.cc:260] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.371973 16430 ts_tablet_manager.cc:1428] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.372828 16434 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.372927 16434 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:16:07.377251 16431 raft_consensus.cc:482] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:07.377318 16431 raft_consensus.cc:504] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.377467 16431 leader_election.cc:288] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667), 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289)
I20260925 15:16:07.377627 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a4a8b553844698992b30977b42958e" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" is_pre_election: true
I20260925 15:16:07.377627 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a4a8b553844698992b30977b42958e" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac" is_pre_election: true
I20260925 15:16:07.377729 16072 raft_consensus.cc:2454] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 0.
I20260925 15:16:07.377730 16212 raft_consensus.cc:2454] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 0.
I20260925 15:16:07.377866 16285 leader_election.cc:302] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46720c1d83f54bd2a9fa0d9ccb38a02c, 4972c60993054eccb391337aefdb127b; no voters:
I20260925 15:16:07.377959 16431 raft_consensus.cc:2781] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:07.378034 16431 raft_consensus.cc:482] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:07.378069 16431 raft_consensus.cc:3037] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.378645 16431 raft_consensus.cc:504] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.378801 16431 leader_election.cc:288] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 election: Requested vote from peers 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667), 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289)
I20260925 15:16:07.378959 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a4a8b553844698992b30977b42958e" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c"
I20260925 15:16:07.378969 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a4a8b553844698992b30977b42958e" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac"
I20260925 15:16:07.379026 16072 raft_consensus.cc:3037] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.379046 16212 raft_consensus.cc:3037] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.379776 16212 raft_consensus.cc:2454] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 1.
I20260925 15:16:07.379789 16072 raft_consensus.cc:2454] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 1.
I20260925 15:16:07.379958 16285 leader_election.cc:302] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46720c1d83f54bd2a9fa0d9ccb38a02c, 4972c60993054eccb391337aefdb127b; no voters:
I20260925 15:16:07.380049 16431 raft_consensus.cc:2781] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:07.380093 16431 raft_consensus.cc:686] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [term 1 LEADER]: Becoming Leader. State: Replica: 4972c60993054eccb391337aefdb127b, State: Running, Role: LEADER
I20260925 15:16:07.380141 16431 consensus_queue.cc:237] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } }
I20260925 15:16:07.380379 16444 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: TxnStatusTablet state changed. Reason: New leader 4972c60993054eccb391337aefdb127b. Latest consensus state: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.380470 16444 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:16:07.380597 16466 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:16:07.380746 16466 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:16:07.380647 15914 catalog_manager.cc:5495] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4972c60993054eccb391337aefdb127b (127.8.43.131). New cstate: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20260925 15:16:07.388006 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.385648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44856 (local address 127.8.43.131:38787)
0925 15:16:07.385718 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.385720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.385729 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:07.385829 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.385830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.385841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.385896 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:07.386021 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.386497 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.387292 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.387493 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.387577 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.387745 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:07.387780 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:07.387850 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.387916 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.387937 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:16:07.388033 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.385593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:38787 (local address 127.0.0.1:44856)
0925 15:16:07.385684 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.385695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.385744 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.385903 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.385904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.385905 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:16:07.385951 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.385953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.386579 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.386580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.387157 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.387159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.387579 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.387580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.387648 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.387662 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:16:07.387850 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:16:07.387859 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.387912 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.387959 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:16:07.388926 16212 raft_consensus.cc:1261] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Refusing update from remote peer 4972c60993054eccb391337aefdb127b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.388944 16072 raft_consensus.cc:1261] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Refusing update from remote peer 4972c60993054eccb391337aefdb127b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.389153 16444 consensus_queue.cc:1035] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [LEADER]: Connected to new peer: Peer: permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.389264 16431 consensus_queue.cc:1035] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b [LEADER]: Connected to new peer: Peer: permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.389734 16432 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: TxnStatusTablet state changed. Reason: New leader 4972c60993054eccb391337aefdb127b. Latest consensus state: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.389822 16432 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:16:07.389887 16434 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: TxnStatusTablet state changed. Reason: New leader 4972c60993054eccb391337aefdb127b. Latest consensus state: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.389973 16434 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:16:07.390511 16444 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.390590 16444 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:16:07.390607 16434 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.390677 16434 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 263c5fc128ac481bb56519c32889c3ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:16:07.390668 16444 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.390734 16444 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 4972c60993054eccb391337aefdb127b: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:16:07.391520 16441 mvcc.cc:205] Tried to move back new op lower bound from 7333271008823660544 to 7333271008789417984. Current Snapshot: MvccSnapshot[applied={T|T < 7333271008823660544}]
I20260925 15:16:07.391671 16432 tablet_replica.cc:439] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } } peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } }
I20260925 15:16:07.391753 16432 tablet_replica.cc:442] T a5a4a8b553844698992b30977b42958e P 46720c1d83f54bd2a9fa0d9ccb38a02c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:16:07.396930 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.394421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:53787 (local address 127.8.43.131:38787)
0925 15:16:07.394456 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.394458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.394550 (+ 92us) server_negotiation.cc:407] Connection header received
0925 15:16:07.394607 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.394607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.394614 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.394654 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:07.394773 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.395191 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.396296 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.396465 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.396517 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.396719 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:07.396741 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:07.396782 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.396855 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.396870 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 15:16:07.397008 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.394343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:38787 (local address 127.8.43.131:53787)
0925 15:16:07.394487 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.394497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.394552 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.394666 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.394667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.394668 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:16:07.394721 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.394722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.395295 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.395296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.396215 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.396217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.396539 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.396540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.396595 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.396609 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:16:07.396800 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:16:07.396802 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.396855 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.396894 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260925 15:16:07.399197 16212 raft_consensus.cc:1261] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Refusing update from remote peer 4972c60993054eccb391337aefdb127b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.399286 16072 raft_consensus.cc:1261] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Refusing update from remote peer 4972c60993054eccb391337aefdb127b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.399402 16444 consensus_queue.cc:1035] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [LEADER]: Connected to new peer: Peer: permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.399498 16431 consensus_queue.cc:1035] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [LEADER]: Connected to new peer: Peer: permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.409901 15915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:07.409998 15915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:07.410643 15915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: Sending DeleteTablet for 3 replicas of tablet ff07b6ddd973454499dc2443edd3046a
I20260925 15:16:07.410883 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet ff07b6ddd973454499dc2443edd3046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:41702
I20260925 15:16:07.410933 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet ff07b6ddd973454499dc2443edd3046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:52258
I20260925 15:16:07.411001 16449 tablet_replica.cc:330] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: stopping tablet replica
I20260925 15:16:07.411062 16449 raft_consensus.cc:2229] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:07.411092 16449 raft_consensus.cc:2258] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.411180 16451 tablet_replica.cc:330] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: stopping tablet replica
I20260925 15:16:07.411270 16451 raft_consensus.cc:2229] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:07.411316 16451 raft_consensus.cc:2258] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.411475 16332 tablet_service.cc:1514] Processing DeleteTablet for tablet ff07b6ddd973454499dc2443edd3046a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:44846
I20260925 15:16:07.411582 16450 tablet_replica.cc:330] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: stopping tablet replica
I20260925 15:16:07.411633 16450 raft_consensus.cc:2229] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:07.411691 16450 raft_consensus.cc:2258] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.411985 16450 ts_tablet_manager.cc:1905] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.413628 16450 ts_tablet_manager.cc:1918] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:16:07.413697 16450 log.cc:1196] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/ff07b6ddd973454499dc2443edd3046a
I20260925 15:16:07.413714 16451 ts_tablet_manager.cc:1905] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.413910 16450 ts_tablet_manager.cc:1939] T ff07b6ddd973454499dc2443edd3046a P 4972c60993054eccb391337aefdb127b: Deleting consensus metadata
I20260925 15:16:07.413934 16449 ts_tablet_manager.cc:1905] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.415117 15901 catalog_manager.cc:4841] TS 4972c60993054eccb391337aefdb127b (127.8.43.131:38787): tablet ff07b6ddd973454499dc2443edd3046a (table test-table [id=ba409b963ab84ac897b24b6a4a0b8ac2]) successfully deleted
I20260925 15:16:07.415186 16451 ts_tablet_manager.cc:1918] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:16:07.415326 16451 log.cc:1196] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/ff07b6ddd973454499dc2443edd3046a
I20260925 15:16:07.415401 16449 ts_tablet_manager.cc:1918] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:16:07.415460 16449 log.cc:1196] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/ff07b6ddd973454499dc2443edd3046a
I20260925 15:16:07.415568 16451 ts_tablet_manager.cc:1939] T ff07b6ddd973454499dc2443edd3046a P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting consensus metadata
I20260925 15:16:07.415678 16449 ts_tablet_manager.cc:1939] T ff07b6ddd973454499dc2443edd3046a P 263c5fc128ac481bb56519c32889c3ac: Deleting consensus metadata
I20260925 15:16:07.415879 15901 catalog_manager.cc:4841] TS 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289): tablet ff07b6ddd973454499dc2443edd3046a (table test-table [id=ba409b963ab84ac897b24b6a4a0b8ac2]) successfully deleted
I20260925 15:16:07.415987 15899 catalog_manager.cc:4841] TS 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667): tablet ff07b6ddd973454499dc2443edd3046a (table test-table [id=ba409b963ab84ac897b24b6a4a0b8ac2]) successfully deleted
Sep 25 15:16:07 dist-test-slave-c5d6 krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:16:07 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:07 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:16:07.436249 15983 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:07.430248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.8.43.190:32955)
0925 15:16:07.430316 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.430318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.430466 (+ 148us) server_negotiation.cc:407] Connection header received
0925 15:16:07.430535 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.430535 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.430544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.430604 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:07.431310 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.431835 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.432944 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.433117 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.433167 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.436184 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49888: BlockingRecv error: recv got EOF from 127.0.0.1:49888 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 15:16:07 dist-test-slave-c5d6 krb5kdc[15870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM
I20260925 15:16:07.444516 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.437063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.8.43.190:32955)
0925 15:16:07.437096 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.437098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.437180 (+ 82us) server_negotiation.cc:407] Connection header received
0925 15:16:07.437251 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.437252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.437262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.437340 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:07.437805 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.438539 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.439187 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.439502 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.439570 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.441978 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:07.441979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:07.441980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:07.441987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:07.443533 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:07.443838 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:07.443839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:07.443839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:07.443896 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:07.444064 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:07.444065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:07.444065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:07.444186 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:07.444278 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.444350 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.444423 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:16:07.446049 15915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:07.446219 15915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:07.447590 16332 tablet_service.cc:1467] Processing CreateTablet for tablet 11c4381c000046818c2426e208f6f711 (DEFAULT_TABLE table=test-table [id=bbf06079eeaf432bbd4b542aedd49e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.447598 16192 tablet_service.cc:1467] Processing CreateTablet for tablet 11c4381c000046818c2426e208f6f711 (DEFAULT_TABLE table=test-table [id=bbf06079eeaf432bbd4b542aedd49e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.447605 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 11c4381c000046818c2426e208f6f711 (DEFAULT_TABLE table=test-table [id=bbf06079eeaf432bbd4b542aedd49e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.447719 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c4381c000046818c2426e208f6f711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.447724 16332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c4381c000046818c2426e208f6f711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.447734 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c4381c000046818c2426e208f6f711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.448964 16427 tablet_bootstrap.cc:492] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: Bootstrap starting.
I20260925 15:16:07.448992 16426 tablet_bootstrap.cc:492] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: Bootstrap starting.
I20260925 15:16:07.448992 16430 tablet_bootstrap.cc:492] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: Bootstrap starting.
I20260925 15:16:07.449307 16426 tablet_bootstrap.cc:654] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.449343 16427 tablet_bootstrap.cc:654] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.449393 16430 tablet_bootstrap.cc:654] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.449940 16426 tablet_bootstrap.cc:492] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: No bootstrap required, opened a new log
I20260925 15:16:07.449986 16426 ts_tablet_manager.cc:1397] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:07.450024 16427 tablet_bootstrap.cc:492] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: No bootstrap required, opened a new log
I20260925 15:16:07.450063 16427 ts_tablet_manager.cc:1397] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:07.450093 16426 raft_consensus.cc:348] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.450165 16426 raft_consensus.cc:374] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.450186 16426 raft_consensus.cc:729] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4972c60993054eccb391337aefdb127b, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.450227 16426 consensus_queue.cc:260] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.450250 16427 raft_consensus.cc:348] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.450316 16427 raft_consensus.cc:374] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.450322 16426 ts_tablet_manager.cc:1428] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.450335 16427 raft_consensus.cc:729] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46720c1d83f54bd2a9fa0d9ccb38a02c, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.450371 16427 consensus_queue.cc:260] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.450444 16427 ts_tablet_manager.cc:1428] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:07.450867 16430 tablet_bootstrap.cc:492] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: No bootstrap required, opened a new log
I20260925 15:16:07.450917 16430 ts_tablet_manager.cc:1397] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:16:07.451066 16430 raft_consensus.cc:348] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.451138 16430 raft_consensus.cc:374] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.451166 16430 raft_consensus.cc:729] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263c5fc128ac481bb56519c32889c3ac, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.451211 16430 consensus_queue.cc:260] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.451362 16430 ts_tablet_manager.cc:1428] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 15:16:07.545001 16432 raft_consensus.cc:482] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:07.545085 16432 raft_consensus.cc:504] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.545403 16432 leader_election.cc:288] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667), 4972c60993054eccb391337aefdb127b (127.8.43.131:38787)
I20260925 15:16:07.548378 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.545673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:50315 (local address 127.8.43.130:43667)
0925 15:16:07.545744 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.545746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.545833 (+ 87us) server_negotiation.cc:407] Connection header received
0925 15:16:07.545909 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.545910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.545919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.545978 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:07.546161 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.546603 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.547672 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.548041 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.548095 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.548123 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.548268 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.548309 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:16:07.548410 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.545740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:34179 (local address 127.8.43.131:38787)
0925 15:16:07.545788 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.545790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.545801 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:16:07.545883 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.545884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.545893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.545970 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:07.546100 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.546602 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.547719 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.548050 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.548139 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.548149 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.548312 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.548349 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:16:07.548404 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.545597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:43667 (local address 127.8.43.129:50315)
0925 15:16:07.545770 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.545781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.545853 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.545994 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.545995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.545997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:07.546080 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.546082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.546701 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.546702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.547591 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.547594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.548118 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.548119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.548193 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.548197 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.548246 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.548300 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":39,"thread_start_us":73,"threads_started":1}
I20260925 15:16:07.548427 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.545597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:38787 (local address 127.8.43.129:34179)
0925 15:16:07.545734 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.545746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.545806 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.545966 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.545967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.545969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:07.546028 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.546030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.546706 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.546707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.547603 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.547605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.548154 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.548155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.548217 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.548220 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.548292 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.548325 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":17,"thread_start_us":73,"threads_started":1}
I20260925 15:16:07.548660 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c4381c000046818c2426e208f6f711" candidate_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac" is_pre_election: true
I20260925 15:16:07.548748 16212 raft_consensus.cc:2454] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46720c1d83f54bd2a9fa0d9ccb38a02c in term 0.
I20260925 15:16:07.548914 16003 leader_election.cc:302] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263c5fc128ac481bb56519c32889c3ac, 46720c1d83f54bd2a9fa0d9ccb38a02c; no voters:
I20260925 15:16:07.548978 16352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c4381c000046818c2426e208f6f711" candidate_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4972c60993054eccb391337aefdb127b" is_pre_election: true
I20260925 15:16:07.549055 16432 raft_consensus.cc:2781] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:07.549106 16432 raft_consensus.cc:482] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:07.549104 16352 raft_consensus.cc:2454] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46720c1d83f54bd2a9fa0d9ccb38a02c in term 0.
I20260925 15:16:07.549136 16432 raft_consensus.cc:3037] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.549834 16432 raft_consensus.cc:504] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.550005 16432 leader_election.cc:288] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [CANDIDATE]: Term 1 election: Requested vote from peers 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667), 4972c60993054eccb391337aefdb127b (127.8.43.131:38787)
I20260925 15:16:07.550161 16352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c4381c000046818c2426e208f6f711" candidate_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4972c60993054eccb391337aefdb127b"
I20260925 15:16:07.550225 16352 raft_consensus.cc:3037] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.550190 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c4381c000046818c2426e208f6f711" candidate_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac"
I20260925 15:16:07.550273 16212 raft_consensus.cc:3037] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.550824 16212 raft_consensus.cc:2454] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46720c1d83f54bd2a9fa0d9ccb38a02c in term 1.
I20260925 15:16:07.550899 16352 raft_consensus.cc:2454] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46720c1d83f54bd2a9fa0d9ccb38a02c in term 1.
I20260925 15:16:07.551030 16003 leader_election.cc:302] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263c5fc128ac481bb56519c32889c3ac, 46720c1d83f54bd2a9fa0d9ccb38a02c; no voters:
I20260925 15:16:07.551110 16432 raft_consensus.cc:2781] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:07.551321 16432 raft_consensus.cc:686] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 LEADER]: Becoming Leader. State: Replica: 46720c1d83f54bd2a9fa0d9ccb38a02c, State: Running, Role: LEADER
I20260925 15:16:07.551396 16432 consensus_queue.cc:237] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.552071 15915 catalog_manager.cc:5495] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c reported cstate change: term changed from 0 to 1, leader changed from <none> to 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129). New cstate: current_term: 1 leader_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } health_report { overall_health: UNKNOWN } } }
I20260925 15:16:07.567850 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.565519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.8.43.129:37289)
0925 15:16:07.565562 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.565564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.565573 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:07.565612 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.565613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.565630 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.565714 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:07.565812 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.566281 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.567030 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.567285 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.567371 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.567538 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:07.567645 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:07.567730 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.567756 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.567776 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:16:07.568622 16212 raft_consensus.cc:1261] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Refusing update from remote peer 46720c1d83f54bd2a9fa0d9ccb38a02c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.568749 16352 raft_consensus.cc:1261] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Refusing update from remote peer 46720c1d83f54bd2a9fa0d9ccb38a02c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.568922 16432 consensus_queue.cc:1035] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [LEADER]: Connected to new peer: Peer: permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.569038 16493 consensus_queue.cc:1035] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.572090 15915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:07.572161 15915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:07.572757 15915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: Sending DeleteTablet for 3 replicas of tablet 11c4381c000046818c2426e208f6f711
I20260925 15:16:07.573011 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c4381c000046818c2426e208f6f711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:41702
I20260925 15:16:07.573050 16332 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c4381c000046818c2426e208f6f711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:44846
I20260925 15:16:07.573041 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c4381c000046818c2426e208f6f711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:52258
I20260925 15:16:07.573097 16449 tablet_replica.cc:330] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: stopping tablet replica
I20260925 15:16:07.573118 16451 tablet_replica.cc:330] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: stopping tablet replica
I20260925 15:16:07.573118 16450 tablet_replica.cc:330] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: stopping tablet replica
I20260925 15:16:07.573143 16449 raft_consensus.cc:2229] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:07.573163 16451 raft_consensus.cc:2229] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:07.573165 16450 raft_consensus.cc:2229] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:07.573195 16449 raft_consensus.cc:2258] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.573210 16450 raft_consensus.cc:2258] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.573251 16451 raft_consensus.cc:2258] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.573501 16451 ts_tablet_manager.cc:1905] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.573766 16449 ts_tablet_manager.cc:1905] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.574829 16451 ts_tablet_manager.cc:1918] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:07.574877 16451 log.cc:1196] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/11c4381c000046818c2426e208f6f711
I20260925 15:16:07.575030 16451 ts_tablet_manager.cc:1939] T 11c4381c000046818c2426e208f6f711 P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting consensus metadata
I20260925 15:16:07.575167 16449 ts_tablet_manager.cc:1918] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:07.575241 16449 log.cc:1196] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/11c4381c000046818c2426e208f6f711
I20260925 15:16:07.575374 15901 catalog_manager.cc:4841] TS 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289): tablet 11c4381c000046818c2426e208f6f711 (table test-table [id=bbf06079eeaf432bbd4b542aedd49e7f]) successfully deleted
I20260925 15:16:07.575453 16449 ts_tablet_manager.cc:1939] T 11c4381c000046818c2426e208f6f711 P 263c5fc128ac481bb56519c32889c3ac: Deleting consensus metadata
I20260925 15:16:07.575774 15899 catalog_manager.cc:4841] TS 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667): tablet 11c4381c000046818c2426e208f6f711 (table test-table [id=bbf06079eeaf432bbd4b542aedd49e7f]) successfully deleted
I20260925 15:16:07.575851 15915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:07.575995 15915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:07.576023 16450 ts_tablet_manager.cc:1905] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.577116 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 9df58437c16447a79facbcc590fe3c6e (DEFAULT_TABLE table=test-table [id=58b3c089abf34ad38d31808315510db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.577226 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df58437c16447a79facbcc590fe3c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.577494 16450 ts_tablet_manager.cc:1918] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:07.577548 16450 log.cc:1196] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/11c4381c000046818c2426e208f6f711
I20260925 15:16:07.577754 16450 ts_tablet_manager.cc:1939] T 11c4381c000046818c2426e208f6f711 P 4972c60993054eccb391337aefdb127b: Deleting consensus metadata
I20260925 15:16:07.578137 16192 tablet_service.cc:1467] Processing CreateTablet for tablet 9df58437c16447a79facbcc590fe3c6e (DEFAULT_TABLE table=test-table [id=58b3c089abf34ad38d31808315510db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.578162 16332 tablet_service.cc:1467] Processing CreateTablet for tablet 9df58437c16447a79facbcc590fe3c6e (DEFAULT_TABLE table=test-table [id=58b3c089abf34ad38d31808315510db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:07.578249 16332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df58437c16447a79facbcc590fe3c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.578254 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df58437c16447a79facbcc590fe3c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:07.578506 16427 tablet_bootstrap.cc:492] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Bootstrap starting.
I20260925 15:16:07.578868 16427 tablet_bootstrap.cc:654] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.579423 16430 tablet_bootstrap.cc:492] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: Bootstrap starting.
I20260925 15:16:07.579495 16426 tablet_bootstrap.cc:492] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: Bootstrap starting.
I20260925 15:16:07.579540 16427 tablet_bootstrap.cc:492] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: No bootstrap required, opened a new log
I20260925 15:16:07.579571 16427 ts_tablet_manager.cc:1397] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:07.579710 16427 raft_consensus.cc:348] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.579775 16426 tablet_bootstrap.cc:654] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.579787 16427 raft_consensus.cc:374] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.579808 16427 raft_consensus.cc:729] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46720c1d83f54bd2a9fa0d9ccb38a02c, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.579856 16427 consensus_queue.cc:260] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.579963 16427 ts_tablet_manager.cc:1428] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.580029 16430 tablet_bootstrap.cc:654] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:07.580092 15901 catalog_manager.cc:4841] TS 4972c60993054eccb391337aefdb127b (127.8.43.131:38787): tablet 11c4381c000046818c2426e208f6f711 (table test-table [id=bbf06079eeaf432bbd4b542aedd49e7f]) successfully deleted
I20260925 15:16:07.580853 16430 tablet_bootstrap.cc:492] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: No bootstrap required, opened a new log
I20260925 15:16:07.580904 16430 ts_tablet_manager.cc:1397] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:16:07.581035 16430 raft_consensus.cc:348] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.581105 16430 raft_consensus.cc:374] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.581126 16430 raft_consensus.cc:729] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263c5fc128ac481bb56519c32889c3ac, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.581167 16430 consensus_queue.cc:260] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.581295 16430 ts_tablet_manager.cc:1428] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 15:16:07.581748 16426 tablet_bootstrap.cc:492] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: No bootstrap required, opened a new log
I20260925 15:16:07.581794 16426 ts_tablet_manager.cc:1397] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:16:07.581881 16426 raft_consensus.cc:348] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.581921 16426 raft_consensus.cc:374] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:07.581933 16426 raft_consensus.cc:729] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4972c60993054eccb391337aefdb127b, State: Initialized, Role: FOLLOWER
I20260925 15:16:07.581984 16426 consensus_queue.cc:260] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.582065 16426 ts_tablet_manager.cc:1428] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.640992 16444 raft_consensus.cc:482] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:07.641126 16444 raft_consensus.cc:504] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.641290 16444 leader_election.cc:288] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289), 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667)
I20260925 15:16:07.641538 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df58437c16447a79facbcc590fe3c6e" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" is_pre_election: true
I20260925 15:16:07.641548 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df58437c16447a79facbcc590fe3c6e" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac" is_pre_election: true
I20260925 15:16:07.641677 16212 raft_consensus.cc:2454] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 0.
I20260925 15:16:07.641690 16071 raft_consensus.cc:2454] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 0.
I20260925 15:16:07.641889 16283 leader_election.cc:302] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263c5fc128ac481bb56519c32889c3ac, 4972c60993054eccb391337aefdb127b; no voters:
I20260925 15:16:07.641993 16444 raft_consensus.cc:2781] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:07.642043 16444 raft_consensus.cc:482] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:07.642062 16444 raft_consensus.cc:3037] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.642709 16444 raft_consensus.cc:504] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.642859 16444 leader_election.cc:288] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 election: Requested vote from peers 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289), 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667)
I20260925 15:16:07.643009 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df58437c16447a79facbcc590fe3c6e" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c"
I20260925 15:16:07.643028 16212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df58437c16447a79facbcc590fe3c6e" candidate_uuid: "4972c60993054eccb391337aefdb127b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263c5fc128ac481bb56519c32889c3ac"
I20260925 15:16:07.643085 16071 raft_consensus.cc:3037] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.643096 16212 raft_consensus.cc:3037] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:07.643716 16071 raft_consensus.cc:2454] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 1.
I20260925 15:16:07.643875 16285 leader_election.cc:302] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46720c1d83f54bd2a9fa0d9ccb38a02c, 4972c60993054eccb391337aefdb127b; no voters:
I20260925 15:16:07.643930 16212 raft_consensus.cc:2454] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4972c60993054eccb391337aefdb127b in term 1.
I20260925 15:16:07.643952 16444 raft_consensus.cc:2781] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:07.643990 16444 raft_consensus.cc:686] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 1 LEADER]: Becoming Leader. State: Replica: 4972c60993054eccb391337aefdb127b, State: Running, Role: LEADER
I20260925 15:16:07.644052 16444 consensus_queue.cc:237] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } }
I20260925 15:16:07.644639 15914 catalog_manager.cc:5495] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4972c60993054eccb391337aefdb127b (127.8.43.131). New cstate: current_term: 1 leader_uuid: "4972c60993054eccb391337aefdb127b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4972c60993054eccb391337aefdb127b" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 38787 } health_report { overall_health: HEALTHY } } }
I20260925 15:16:07.658118 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.655679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.8.43.131:38787)
0925 15:16:07.655715 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.655717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.655725 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:16:07.655782 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.655782 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.655789 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.655832 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:07.655944 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.656400 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.657452 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.657625 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.657675 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.657876 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:07.657906 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:07.657960 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.658046 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.658063 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:16:07.660416 16212 raft_consensus.cc:1261] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Refusing update from remote peer 4972c60993054eccb391337aefdb127b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.660570 16071 raft_consensus.cc:1261] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Refusing update from remote peer 4972c60993054eccb391337aefdb127b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:07.660645 16436 consensus_queue.cc:1035] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [LEADER]: Connected to new peer: Peer: permanent_uuid: "263c5fc128ac481bb56519c32889c3ac" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.660779 16436 consensus_queue.cc:1035] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [LEADER]: Connected to new peer: Peer: permanent_uuid: "46720c1d83f54bd2a9fa0d9ccb38a02c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:07.672147 15914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:07.672286 15914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:07.672930 15914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34c120ebf3f347daaccdf201f2811801: Sending DeleteTablet for 3 replicas of tablet 9df58437c16447a79facbcc590fe3c6e
I20260925 15:16:07.673175 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df58437c16447a79facbcc590fe3c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:41702
I20260925 15:16:07.673207 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df58437c16447a79facbcc590fe3c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:52258
I20260925 15:16:07.673264 16449 tablet_replica.cc:330] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: stopping tablet replica
I20260925 15:16:07.673324 16449 raft_consensus.cc:2229] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:07.673362 16449 raft_consensus.cc:2258] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.673382 16451 tablet_replica.cc:330] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: stopping tablet replica
I20260925 15:16:07.673441 16451 raft_consensus.cc:2229] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:07.673481 16451 raft_consensus.cc:2258] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.673687 16449 ts_tablet_manager.cc:1905] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.673849 16451 ts_tablet_manager.cc:1905] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.675315 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 15987
I20260925 15:16:07.675452 16449 ts_tablet_manager.cc:1918] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:16:07.675453 16451 ts_tablet_manager.cc:1918] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:16:07.675503 16449 log.cc:1196] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/9df58437c16447a79facbcc590fe3c6e
I20260925 15:16:07.675503 16451 log.cc:1196] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/9df58437c16447a79facbcc590fe3c6e
I20260925 15:16:07.675753 16449 ts_tablet_manager.cc:1939] T 9df58437c16447a79facbcc590fe3c6e P 263c5fc128ac481bb56519c32889c3ac: Deleting consensus metadata
I20260925 15:16:07.675755 16451 ts_tablet_manager.cc:1939] T 9df58437c16447a79facbcc590fe3c6e P 46720c1d83f54bd2a9fa0d9ccb38a02c: Deleting consensus metadata
I20260925 15:16:07.676734 16332 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df58437c16447a79facbcc590fe3c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:07 UTC) from {username='oryx', principal='oryx/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:44846
I20260925 15:16:07.676837 16450 tablet_replica.cc:330] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: stopping tablet replica
I20260925 15:16:07.676908 16450 raft_consensus.cc:2229] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:07.676975 16450 raft_consensus.cc:2258] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:07.677223 16450 ts_tablet_manager.cc:1905] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:07.678503 16450 ts_tablet_manager.cc:1918] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:16:07.678571 16450 log.cc:1196] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/9df58437c16447a79facbcc590fe3c6e
I20260925 15:16:07.678819 16450 ts_tablet_manager.cc:1939] T 9df58437c16447a79facbcc590fe3c6e P 4972c60993054eccb391337aefdb127b: Deleting consensus metadata
I20260925 15:16:07.679134 15901 catalog_manager.cc:4841] TS 4972c60993054eccb391337aefdb127b (127.8.43.131:38787): tablet 9df58437c16447a79facbcc590fe3c6e (table test-table [id=58b3c089abf34ad38d31808315510db0]) successfully deleted
W20260925 15:16:07.682379 15901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:37289 (error 108)
I20260925 15:16:07.682423 15899 catalog_manager.cc:4841] TS 263c5fc128ac481bb56519c32889c3ac (127.8.43.130:43667): tablet 9df58437c16447a79facbcc590fe3c6e (table test-table [id=58b3c089abf34ad38d31808315510db0]) successfully deleted
I20260925 15:16:07.682971 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.682780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:37289 (local address 127.0.0.1:52286)
0925 15:16:07.682873 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.682921 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:37289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":37,"threads_started":1}
W20260925 15:16:07.683071 15901 catalog_manager.cc:4584] TS 46720c1d83f54bd2a9fa0d9ccb38a02c (127.8.43.129:37289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9df58437c16447a79facbcc590fe3c6e: Network error: Client connection negotiation failed: client connection to 127.8.43.129:37289: connect: Connection refused (error 111)
I20260925 15:16:07.683135 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 16127
I20260925 15:16:07.688431 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 16267
I20260925 15:16:07.694232 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 15883
2026-09-25T15:16:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2650 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:16:07 dist-test-slave-c5d6 krb5kdc[16510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:16:07 dist-test-slave-c5d6 krb5kdc[16510](info): set up 2 sockets
Sep 25 15:16:07 dist-test-slave-c5d6 krb5kdc[16510](info): commencing operation
krb5kdc: starting...
W20260925 15:16:08.731714 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:16:08 dist-test-slave-c5d6 krb5kdc[16510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:16:08.846007 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:37967
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:37967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:08.925189 16523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:08.925392 16523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:08.925415 16523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:08.926473 16523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:08.926509 16523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:08.926523 16523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:08.926535 16523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:08.926551 16523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:08.927971 16523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:37967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:37967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:08.928181 16523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:08.928354 16523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:08.933189 16528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:08.933190 16529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:08.933190 16531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:08.933362 16523 server_base.cc:1034] running on GCE node
I20260925 15:16:08.933683 16523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:08.934003 16523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:08.935145 16523 hybrid_clock.cc:648] HybridClock initialized: now 1790349368935126 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T15:16:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1248 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:16:08 dist-test-slave-c5d6 krb5kdc[16536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:16:08 dist-test-slave-c5d6 krb5kdc[16536](info): set up 2 sockets
Sep 25 15:16:08 dist-test-slave-c5d6 krb5kdc[16536](info): commencing operation
krb5kdc: starting...
W20260925 15:16:09.983810 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:16:10.097539 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:42519
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:10.180670 16549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:10.180886 16549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:10.180912 16549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:10.182027 16549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:10.182063 16549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:10.182080 16549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:10.182093 16549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:10.182107 16549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:10.183485 16549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:10.183676 16549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:10.183838 16549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:10.188822 16554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.188854 16549 server_base.cc:1034] running on GCE node
W20260925 15:16:10.188822 16557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.188828 16555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.189248 16549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:10.189550 16549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:10.190716 16549 hybrid_clock.cc:648] HybridClock initialized: now 1790349370190704 us; error 28 us; skew 500 ppm
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:10.197539 16549 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:16:10.197981 16549 webserver.cc:460] Webserver started at http://127.8.43.190:41381/ using document root <none> and password file <none>
I20260925 15:16:10.198232 16549 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:10.198279 16549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:10.198462 16549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:10.199579 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "184950f1f33a43d389f3cc3a215a6cbc"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "4adae2c8f9fff2417fe5a57584b2eba0"
server_key_iv: "5e36f4d227c7527b100414a334a8f506"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.199952 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "184950f1f33a43d389f3cc3a215a6cbc"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "4adae2c8f9fff2417fe5a57584b2eba0"
server_key_iv: "5e36f4d227c7527b100414a334a8f506"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.201256 16549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:16:10.201972 16564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.202140 16549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:10.202262 16549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "184950f1f33a43d389f3cc3a215a6cbc"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "4adae2c8f9fff2417fe5a57584b2eba0"
server_key_iv: "5e36f4d227c7527b100414a334a8f506"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.202335 16549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:10.224212 16549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:10.226048 16549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:10.226214 16549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:10.230079 16549 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:42519
I20260925 15:16:10.230115 16616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:42519 every 8 connection(s)
I20260925 15:16:10.230492 16549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:10.231462 16617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:10.231806 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 16549
I20260925 15:16:10.231914 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:16:10.232112 8366 external_mini_cluster.cc:1428] Setting key 60f0c8e2d3d5d86b55cf8f5fae98c18a
I20260925 15:16:10.234516 16617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Bootstrap starting.
I20260925 15:16:10.235265 16617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:10.235553 16617 log.cc:824] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:10.236290 16617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: No bootstrap required, opened a new log
I20260925 15:16:10.238420 16617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } }
I20260925 15:16:10.238569 16617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:10.238617 16617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184950f1f33a43d389f3cc3a215a6cbc, State: Initialized, Role: FOLLOWER
I20260925 15:16:10.238744 16617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } }
I20260925 15:16:10.238824 16617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:10.238852 16617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:10.238897 16617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 0 FOLLOWER]: Advancing to term 1
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:16:10.239542 16617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } }
I20260925 15:16:10.239657 16617 leader_election.cc:302] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184950f1f33a43d389f3cc3a215a6cbc; no voters:
I20260925 15:16:10.239856 16617 leader_election.cc:288] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:16:10.239885 16622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:10.239995 16622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 1 LEADER]: Becoming Leader. State: Replica: 184950f1f33a43d389f3cc3a215a6cbc, State: Running, Role: LEADER
I20260925 15:16:10.240104 16617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:10.240087 16622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } }
I20260925 15:16:10.240480 16623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184950f1f33a43d389f3cc3a215a6cbc. Latest consensus state: current_term: 1 leader_uuid: "184950f1f33a43d389f3cc3a215a6cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } } }
I20260925 15:16:10.240566 16623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:10.240890 16629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:10.241602 16629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:10.240638 16624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "184950f1f33a43d389f3cc3a215a6cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } } }
I20260925 15:16:10.242146 16624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:10.243352 16629 catalog_manager.cc:1347] Generated new cluster ID: a2dd09b52731452dbc724ca34a5c3a4d
I20260925 15:16:10.243429 16629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:10.243594 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.232650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.8.43.190:42519)
0925 15:16:10.232848 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.232859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.232893 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:16:10.233429 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.233439 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.233507 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.233654 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:10.234230 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.235647 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.236569 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.237206 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.237318 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.239690 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:10.239694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:10.239694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:10.239705 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:10.241927 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.242383 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.242385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.242386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.242463 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.242692 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.242693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.242693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.242975 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:10.243128 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.243278 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.243407 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260925 15:16:10.263578 16629 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:10.264315 16629 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:10.276731 16629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Generated new TSK 0
I20260925 15:16:10.276923 16629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:16:10.305819 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:42519
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:10.386582 16643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:10.386772 16643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:10.386795 16643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:10.388015 16643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:10.388055 16643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:10.388106 16643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:16:10.389518 16643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:10.389765 16643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:10.390003 16643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:10.394654 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.394794 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.394882 16643 server_base.cc:1034] running on GCE node
W20260925 15:16:10.394872 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.395156 16643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:10.395435 16643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:10.396639 16643 hybrid_clock.cc:648] HybridClock initialized: now 1790349370396621 us; error 29 us; skew 500 ppm
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:10.399118 16643 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:16:10.399572 16643 webserver.cc:460] Webserver started at http://127.8.43.129:42337/ using document root <none> and password file <none>
I20260925 15:16:10.399832 16643 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:10.399905 16643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:10.400050 16643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:10.401098 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "59b475f1e51f4e48bc9190c8dc894225"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "4d1dbdc449d86003ac75a772fce0ff76"
server_key_iv: "af34556b1ec620537a9879e2f7ab1d3f"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.401458 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "59b475f1e51f4e48bc9190c8dc894225"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "4d1dbdc449d86003ac75a772fce0ff76"
server_key_iv: "af34556b1ec620537a9879e2f7ab1d3f"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.402763 16643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:16:10.403488 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.403687 16643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:16:10.403810 16643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "59b475f1e51f4e48bc9190c8dc894225"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "4d1dbdc449d86003ac75a772fce0ff76"
server_key_iv: "af34556b1ec620537a9879e2f7ab1d3f"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.403885 16643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:10.426853 16643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:10.429278 16643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:10.429446 16643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:10.429711 16643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:10.430042 16643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:10.430096 16643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.430136 16643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:10.430166 16643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.435889 16643 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:38131
I20260925 15:16:10.435981 16771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:38131 every 8 connection(s)
I20260925 15:16:10.436290 16643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:10.440611 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 16643
I20260925 15:16:10.440726 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:16:10.440964 8366 external_mini_cluster.cc:1428] Setting key 673797ee63f24a29865f8d58d6cad55c
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:16:10.446422 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.437037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:37905 (local address 127.8.43.190:42519)
0925 15:16:10.437084 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.437086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.437308 (+ 222us) server_negotiation.cc:407] Connection header received
0925 15:16:10.437774 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.437775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.437784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.437827 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:10.438701 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.439184 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.440300 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.440577 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.440644 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.443433 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:10.443435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:10.443435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:10.443443 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:10.444917 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.445445 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.445446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.445446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.445496 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.445777 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.445778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.445778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.445877 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:10.445928 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.446291 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.446358 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:16:10.446751 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.437028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:37905)
0925 15:16:10.437187 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:10.437222 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:10.437693 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:10.437880 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:10.437883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:10.438428 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:10.438616 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.438620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.439309 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.439310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.440229 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.440231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.440652 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.440654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.440836 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.441538 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:10.441582 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:10.443343 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:10.445003 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.445009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.445011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.445369 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.445565 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.445566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.445567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.445710 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.445944 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:10.445951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:10.446134 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 15:16:10.446264 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:16:10.446459 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260925 15:16:10.447331 16772 heartbeater.cc:347] Connected to a master server at 127.8.43.190:42519
I20260925 15:16:10.447438 16772 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:10.447639 16772 heartbeater.cc:511] Master 127.8.43.190:42519 requested a full tablet report, sending...
I20260925 15:16:10.448165 16581 ts_manager.cc:194] Registered new tserver with Master: 59b475f1e51f4e48bc9190c8dc894225 (127.8.43.129:38131)
I20260925 15:16:10.448781 16581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:37905
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:16:10.466181 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:42519
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:10.547945 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:10.548161 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:10.548194 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:10.549324 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:10.549386 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:10.549427 16777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:16:10.550771 16777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:10.551015 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:10.551246 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:10.555935 16782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.555962 16783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.556130 16777 server_base.cc:1034] running on GCE node
W20260925 15:16:10.555944 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.556461 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:10.556670 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:10.557848 16777 hybrid_clock.cc:648] HybridClock initialized: now 1790349370557837 us; error 43 us; skew 500 ppm
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:10.560387 16777 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:16:10.560775 16777 webserver.cc:460] Webserver started at http://127.8.43.130:45283/ using document root <none> and password file <none>
I20260925 15:16:10.561025 16777 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:10.561091 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:10.561244 16777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:10.562242 16777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "cb787b4a5ccc46798a8c673eef640cc2"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "7932ea37b02abb0457113e2fd779f5f0"
server_key_iv: "94682264a5fe6358e47602435df87033"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.562597 16777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "cb787b4a5ccc46798a8c673eef640cc2"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "7932ea37b02abb0457113e2fd779f5f0"
server_key_iv: "94682264a5fe6358e47602435df87033"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.563984 16777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:16:10.564702 16792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.564862 16777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:16:10.564956 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "cb787b4a5ccc46798a8c673eef640cc2"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "7932ea37b02abb0457113e2fd779f5f0"
server_key_iv: "94682264a5fe6358e47602435df87033"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.565040 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:10.592480 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:10.594386 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:10.594543 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:10.594794 16777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:10.595085 16777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:10.595137 16777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.595178 16777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:10.595203 16777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.600958 16777 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:40011
I20260925 15:16:10.601001 16905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:40011 every 8 connection(s)
I20260925 15:16:10.601305 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:16:10.610541 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.601971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:43207 (local address 127.8.43.190:42519)
0925 15:16:10.602045 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.602047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.602227 (+ 180us) server_negotiation.cc:407] Connection header received
0925 15:16:10.602820 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.602821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.602834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.602897 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:10.603890 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.604435 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.605504 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.605707 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.605790 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.607850 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:10.607851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:10.607852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:10.607859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:10.609022 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.609532 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.609533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.609534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.609593 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.609860 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.609861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.609861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.609972 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:10.610074 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.610350 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.610444 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:16:10.610705 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.601924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:43207)
0925 15:16:10.602104 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:10.602139 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:10.602648 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:10.602968 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:10.602972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:10.603588 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:10.603796 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.603801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.604517 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.604518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.605403 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.605405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.605798 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.605799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.605929 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.606347 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:10.606355 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:10.607751 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:10.609126 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.609131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.609133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.609434 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.609664 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.609664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.609665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.609788 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.610065 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:10.610071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:10.610206 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 15:16:10.610327 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:16:10.610472 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 15:16:10.610912 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 16777
I20260925 15:16:10.611028 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:16:10.611258 8366 external_mini_cluster.cc:1428] Setting key 5318c01d9a00912e7d3b1405fd53dfda
I20260925 15:16:10.611284 16906 heartbeater.cc:347] Connected to a master server at 127.8.43.190:42519
I20260925 15:16:10.611375 16906 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:10.611567 16906 heartbeater.cc:511] Master 127.8.43.190:42519 requested a full tablet report, sending...
I20260925 15:16:10.611943 16581 ts_manager.cc:194] Registered new tserver with Master: cb787b4a5ccc46798a8c673eef640cc2 (127.8.43.130:40011)
I20260925 15:16:10.612432 16581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:43207
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:16:10.634042 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:42519
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:10.716339 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:10.716533 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:10.716558 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:10.717746 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:10.717789 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:10.717818 16911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:16:10.719192 16911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:10.719465 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:10.719703 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:10.724634 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.724701 16911 server_base.cc:1034] running on GCE node
W20260925 15:16:10.724640 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.724640 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.725121 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:10.725368 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:10.726545 16911 hybrid_clock.cc:648] HybridClock initialized: now 1790349370726533 us; error 25 us; skew 500 ppm
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:10.729072 16911 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:16:10.729475 16911 webserver.cc:460] Webserver started at http://127.8.43.131:35435/ using document root <none> and password file <none>
I20260925 15:16:10.729694 16911 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:10.729765 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:10.729912 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:10.730928 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "8ab5793ee1ac499d9a06a49940157ef6"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "77f359e254fbf77e2b4a73e71e26b95a"
server_key_iv: "b8d30fdb7ff53f2791fd2869d2986bae"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.731336 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "8ab5793ee1ac499d9a06a49940157ef6"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "77f359e254fbf77e2b4a73e71e26b95a"
server_key_iv: "b8d30fdb7ff53f2791fd2869d2986bae"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.732590 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:10.733294 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.733487 16911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:10.733582 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "8ab5793ee1ac499d9a06a49940157ef6"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "77f359e254fbf77e2b4a73e71e26b95a"
server_key_iv: "b8d30fdb7ff53f2791fd2869d2986bae"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.733664 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:10.753191 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:10.755517 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:10.755666 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:10.755899 16911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:10.756191 16911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:10.756238 16911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.756276 16911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:10.756318 16911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.761909 16911 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:37209
I20260925 15:16:10.761978 17039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:37209 every 8 connection(s)
I20260925 15:16:10.762305 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:16:10.768760 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 16911
I20260925 15:16:10.768865 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:16:10.769053 8366 external_mini_cluster.cc:1428] Setting key 5dd973c87ed1dd54016059cd340c9370
I20260925 15:16:10.771595 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.763003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:59129 (local address 127.8.43.190:42519)
0925 15:16:10.763051 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.763053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.763308 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:16:10.763817 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.763818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.763828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.763889 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:10.764820 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.765317 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.766268 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.766461 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.766509 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.768650 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:10.768652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:10.768652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:10.768660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:10.770286 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.770662 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.770663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.770663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.770719 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.771010 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.771011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.771011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.771114 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:10.771167 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.771468 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.771532 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:16:10.771895 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.762994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:59129)
0925 15:16:10.763147 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:10.763181 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:10.763707 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:10.763966 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:10.763969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:10.764535 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:10.764727 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.764731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.765390 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.765391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.766201 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.766203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.766527 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.766528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.766622 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.767170 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:10.767177 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:10.768552 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:10.770352 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.770357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.770359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.770567 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.770837 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.770838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.770838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.770943 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.771157 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:10.771160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:10.771355 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 15:16:10.771442 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:16:10.771560 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260925 15:16:10.772455 17040 heartbeater.cc:347] Connected to a master server at 127.8.43.190:42519
I20260925 15:16:10.772563 17040 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:10.772742 17040 heartbeater.cc:511] Master 127.8.43.190:42519 requested a full tablet report, sending...
I20260925 15:16:10.773118 16581 ts_manager.cc:194] Registered new tserver with Master: 8ab5793ee1ac499d9a06a49940157ef6 (127.8.43.131:37209)
I20260925 15:16:10.773491 16581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:59129
I20260925 15:16:10.780933 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab.
Entry for principal oryx/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab.
WARNING: no policy specified for oryx/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab.
Entry for principal oryx/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab.
WARNING: no policy specified for oryx/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab.
Entry for principal oryx/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab.
I20260925 15:16:10.846906 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 16643
I20260925 15:16:10.851361 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 16777
I20260925 15:16:10.855787 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 16911
I20260925 15:16:10.859753 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 16549
I20260925 15:16:10.864713 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:42519
--webserver_interface=127.8.43.190
--webserver_port=41381
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:10.943035 17049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:10.943279 17049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:10.943325 17049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:10.944450 17049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:10.944516 17049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:10.944540 17049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:10.944561 17049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:10.944589 17049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:10.946014 17049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=41381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:10.946276 17049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:10.946507 17049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:10.951102 17054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.951081 17057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.951138 17055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.951458 17049 server_base.cc:1034] running on GCE node
I20260925 15:16:10.951682 17049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:10.951921 17049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:10.953058 17049 hybrid_clock.cc:648] HybridClock initialized: now 1790349370953048 us; error 30 us; skew 500 ppm
Sep 25 15:16:10 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:10.955466 17049 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:16:10.955840 17049 webserver.cc:460] Webserver started at http://127.8.43.190:41381/ using document root <none> and password file <none>
I20260925 15:16:10.956027 17049 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:10.956073 17049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:10.957297 17049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:10.957895 17064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.958065 17049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:10.958168 17049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "184950f1f33a43d389f3cc3a215a6cbc"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "4adae2c8f9fff2417fe5a57584b2eba0"
server_key_iv: "5e36f4d227c7527b100414a334a8f506"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.958508 17049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:10.969674 17049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:10.971997 17049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:10.972157 17049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:10.975874 17049 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:42519
I20260925 15:16:10.975919 17116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:42519 every 8 connection(s)
I20260925 15:16:10.976310 17049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:10.977766 17117 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:16:10.978405 17117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Bootstrap starting.
I20260925 15:16:10.978794 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 17049
I20260925 15:16:10.979176 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:38131
--local_ip_for_outbound_sockets=127.8.43.129
--tserver_master_addrs=127.8.43.190:42519
--webserver_port=42337
--webserver_interface=127.8.43.129
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.43.129
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:16:10.986716 17117 log.cc:824] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:10.988438 17117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:16:10.988727 17117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Bootstrap complete.
I20260925 15:16:10.990885 17117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } }
I20260925 15:16:10.991097 17117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184950f1f33a43d389f3cc3a215a6cbc, State: Initialized, Role: FOLLOWER
I20260925 15:16:10.991284 17117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } }
I20260925 15:16:10.991360 17117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:10.991410 17117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:10.991458 17117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:16:10.992406 17117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } }
I20260925 15:16:10.992511 17117 leader_election.cc:302] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184950f1f33a43d389f3cc3a215a6cbc; no voters:
I20260925 15:16:10.992686 17117 leader_election.cc:288] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:16:10.992722 17121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:16:10.992950 17117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:10.992950 17121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [term 2 LEADER]: Becoming Leader. State: Replica: 184950f1f33a43d389f3cc3a215a6cbc, State: Running, Role: LEADER
I20260925 15:16:10.993072 17121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } }
I20260925 15:16:10.993320 17122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "184950f1f33a43d389f3cc3a215a6cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } } }
I20260925 15:16:10.993412 17122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:10.993374 17123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184950f1f33a43d389f3cc3a215a6cbc. Latest consensus state: current_term: 2 leader_uuid: "184950f1f33a43d389f3cc3a215a6cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184950f1f33a43d389f3cc3a215a6cbc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42519 } } }
I20260925 15:16:10.993608 17123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:10.993606 17128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:10.994114 17128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:10.994273 17128 catalog_manager.cc:1259] Loaded cluster ID: a2dd09b52731452dbc724ca34a5c3a4d
I20260925 15:16:10.994462 17128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:10.995368 17128 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:10.995559 17128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 184950f1f33a43d389f3cc3a215a6cbc: Loaded TSK: 0
I20260925 15:16:10.995775 17128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:16:11.075874 17119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:11.076063 17119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:11.076086 17119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:11.077237 17119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:11.077277 17119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:11.077303 17119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:16:11.078696 17119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.129.keytab
--principal=oryx/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:38131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=42337
--tserver_master_addrs=127.8.43.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:11.078913 17119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:11.079151 17119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:11.083881 17144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:11.083986 17119 server_base.cc:1034] running on GCE node
W20260925 15:16:11.083849 17143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:11.083969 17146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:11.084388 17119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:11.084652 17119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:11.085803 17119 hybrid_clock.cc:648] HybridClock initialized: now 1790349371085793 us; error 32 us; skew 500 ppm
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349371, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:11.088224 17119 init.cc:377] Logged in from keytab as oryx/127.8.43.129@KRBTEST.COM (short username oryx)
I20260925 15:16:11.088697 17119 webserver.cc:460] Webserver started at http://127.8.43.129:42337/ using document root <none> and password file <none>
I20260925 15:16:11.088919 17119 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:11.088985 17119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:11.090273 17119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:11.090906 17153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.091090 17119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:11.091212 17119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "59b475f1e51f4e48bc9190c8dc894225"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "4d1dbdc449d86003ac75a772fce0ff76"
server_key_iv: "af34556b1ec620537a9879e2f7ab1d3f"
server_key_version: "encryptionkey@0"
I20260925 15:16:11.091645 17119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:11.131170 17119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:11.133512 17119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:11.133661 17119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:11.134019 17119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:11.134379 17119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:11.134423 17119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.134455 17119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:11.134473 17119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.141804 17119 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:38131
I20260925 15:16:11.141842 17266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:38131 every 8 connection(s)
I20260925 15:16:11.142185 17119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:11.145020 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 17119
I20260925 15:16:11.145310 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:40011
--local_ip_for_outbound_sockets=127.8.43.130
--tserver_master_addrs=127.8.43.190:42519
--webserver_port=45283
--webserver_interface=127.8.43.130
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.43.130
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:11.151657 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.142892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:38535)
0925 15:16:11.143172 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.143213 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.143725 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.144159 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.144163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.144713 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.144886 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.144890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.146115 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.146116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.147291 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.147294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.148038 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.148039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.148164 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.148654 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.148662 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.151403 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
W20260925 15:16:11.151665 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.143011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:38535 (local address 127.8.43.190:42519)
0925 15:16:11.143213 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.143245 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.143319 (+ 74us) server_negotiation.cc:407] Connection header received
0925 15:16:11.143948 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.143953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.143992 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.144119 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.145000 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.146018 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.147398 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.147937 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.148058 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.151526 (+ 3468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:38535: BlockingRecv error: recv got EOF from 127.8.43.129:38535 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
W20260925 15:16:11.151922 17267 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:42519 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.43.190:42519: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:11.156841 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.152166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:57189)
0925 15:16:11.152213 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.152223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.152261 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.152433 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.152434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.152543 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.152589 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.152590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.153543 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.153544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.154039 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.154041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.154541 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.154541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.154585 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.155014 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.155018 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.156798 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:16:11.156956 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.152251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:57189 (local address 127.8.43.190:42519)
0925 15:16:11.152306 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.152308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.152317 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:11.152342 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.152343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.152357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.152426 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.152658 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.153458 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.154117 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.154463 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.154542 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.156899 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:57189: BlockingRecv error: recv got EOF from 127.8.43.129:57189 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:11.161127 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.157098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:42247)
0925 15:16:11.157187 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.157208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.157269 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.157393 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.157405 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.157532 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.157555 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.157556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.158189 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.158190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.158660 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.158662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.159003 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.159003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.159060 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.159443 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.159447 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.161030 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 15:16:11.161147 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.157211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:42247 (local address 127.8.43.190:42519)
0925 15:16:11.157268 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.157270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.157279 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:11.157310 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.157311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.157321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.157381 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.157627 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.158094 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.158743 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.158923 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.158993 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.161095 (+ 2102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:42247: BlockingRecv error: recv got EOF from 127.8.43.129:42247 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 15:16:11.161257 17267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:16:11.230299 17271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:11.230484 17271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:11.230506 17271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:11.231702 17271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:11.231740 17271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:11.231766 17271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:16:11.233183 17271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.130.keytab
--principal=oryx/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:40011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=45283
--tserver_master_addrs=127.8.43.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:11.233400 17271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:11.233629 17271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:11.238587 17279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:11.238611 17276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:11.238624 17277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:11.238775 17271 server_base.cc:1034] running on GCE node
I20260925 15:16:11.239063 17271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:11.239326 17271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:11.240492 17271 hybrid_clock.cc:648] HybridClock initialized: now 1790349371240480 us; error 27 us; skew 500 ppm
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349371, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:11.243027 17271 init.cc:377] Logged in from keytab as oryx/127.8.43.130@KRBTEST.COM (short username oryx)
I20260925 15:16:11.243538 17271 webserver.cc:460] Webserver started at http://127.8.43.130:45283/ using document root <none> and password file <none>
I20260925 15:16:11.243768 17271 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:11.243825 17271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:11.245157 17271 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:11.245714 17286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.245883 17271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:11.245985 17271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "cb787b4a5ccc46798a8c673eef640cc2"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "7932ea37b02abb0457113e2fd779f5f0"
server_key_iv: "94682264a5fe6358e47602435df87033"
server_key_version: "encryptionkey@0"
I20260925 15:16:11.246333 17271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:11.267869 17271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:11.269955 17271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:11.270128 17271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:11.270380 17271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:11.270694 17271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:11.270778 17271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.270815 17271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:11.270843 17271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.276480 17271 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:40011
I20260925 15:16:11.276553 17399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:40011 every 8 connection(s)
I20260925 15:16:11.276940 17271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:11.280843 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 17271
I20260925 15:16:11.281158 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:37209
--local_ip_for_outbound_sockets=127.8.43.131
--tserver_master_addrs=127.8.43.190:42519
--webserver_port=35435
--webserver_interface=127.8.43.131
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.43.131
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:16:11.285058 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.277783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:36697 (local address 127.8.43.190:42519)
0925 15:16:11.277845 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.277848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.278091 (+ 243us) server_negotiation.cc:407] Connection header received
0925 15:16:11.278631 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.278632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.278647 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.278692 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.279738 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.280223 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.281335 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.281523 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.281555 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.285011 (+ 3456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:36697: BlockingRecv error: recv got EOF from 127.8.43.130:36697 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:16:11.285187 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.277781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:36697)
0925 15:16:11.277970 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.278006 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.278523 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.278764 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.278768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.279429 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.279627 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.279632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.280313 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.280315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.281211 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.281214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.281625 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.281626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.281727 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.282264 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.282275 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.284942 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
W20260925 15:16:11.285491 17400 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:42519 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.43.190:42519: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:11.290400 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.285726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:54629)
0925 15:16:11.285781 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.285791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.285830 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.285960 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.285961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.286120 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.286178 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.286180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.286763 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.286764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.287738 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.287741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.288063 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.288064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.288141 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.288513 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.288519 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.290320 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:16:11.290452 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.285793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:54629 (local address 127.8.43.190:42519)
0925 15:16:11.285842 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.285844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.285857 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:16:11.285880 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.285881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.285888 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.285934 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.286256 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.286689 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.287817 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.287989 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.288039 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.290389 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:54629: BlockingRecv error: recv got EOF from 127.8.43.130:54629 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:11.294826 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.290632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:57267)
0925 15:16:11.290754 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.290764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.290808 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.290950 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.290950 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.291066 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.291090 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.291091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.291751 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.291752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.292229 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.292231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.292573 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.292574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.292630 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.293088 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.293093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.294758 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
W20260925 15:16:11.294939 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.290673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:57267 (local address 127.8.43.190:42519)
0925 15:16:11.290707 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.290709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.290809 (+ 100us) server_negotiation.cc:407] Connection header received
0925 15:16:11.290867 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.290868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.290877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.290947 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.291160 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.291657 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.292307 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.292492 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.292572 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.294866 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:57267: BlockingRecv error: recv got EOF from 127.8.43.130:57267 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 15:16:11.295120 17400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:16:11.364130 17403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:11.364357 17403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:11.364395 17403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:11.365624 17403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:11.365688 17403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:11.365733 17403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:16:11.367169 17403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/krb5kdc/oryx_127.8.43.131.keytab
--principal=oryx/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:37209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=35435
--tserver_master_addrs=127.8.43.190:42519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:11.367446 17403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:11.367683 17403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:11.372499 17410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:11.372645 17409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:11.372856 17403 server_base.cc:1034] running on GCE node
W20260925 15:16:11.372753 17412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:11.373142 17403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:11.373411 17403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:11.374575 17403 hybrid_clock.cc:648] HybridClock initialized: now 1790349371374563 us; error 42 us; skew 500 ppm
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349371, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:11.377106 17403 init.cc:377] Logged in from keytab as oryx/127.8.43.131@KRBTEST.COM (short username oryx)
I20260925 15:16:11.377490 17403 webserver.cc:460] Webserver started at http://127.8.43.131:35435/ using document root <none> and password file <none>
I20260925 15:16:11.377687 17403 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:11.377739 17403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:11.378980 17403 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:11.379576 17419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.379760 17403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.379846 17403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "8ab5793ee1ac499d9a06a49940157ef6"
format_stamp: "Formatted at 2026-09-25 15:16:10 on dist-test-slave-c5d6"
server_key: "77f359e254fbf77e2b4a73e71e26b95a"
server_key_iv: "b8d30fdb7ff53f2791fd2869d2986bae"
server_key_version: "encryptionkey@0"
I20260925 15:16:11.380165 17403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:11.397046 17403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:11.399590 17403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:11.399698 17403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:11.399879 17403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:11.400123 17403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:11.400151 17403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.400173 17403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:11.400185 17403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:11.405622 17403 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:37209
I20260925 15:16:11.405745 17532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:37209 every 8 connection(s)
I20260925 15:16:11.406049 17403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:11.406430 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 17403
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:16:11.413882 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.407117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:49175 (local address 127.8.43.190:42519)
0925 15:16:11.407237 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.407239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.407251 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:16:11.407762 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.407762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.407771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.407852 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.408779 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.409244 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.410197 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.410408 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.410496 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.413838 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:49175: BlockingRecv error: recv got EOF from 127.8.43.131:49175 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:16:11.414081 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.406797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:49175)
0925 15:16:11.406993 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.407029 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.407663 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.407938 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.407942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.408496 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.408679 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.408683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.409334 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.409335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.410106 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.410108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.410509 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.410510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.410643 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.411142 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.411151 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.413779 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
W20260925 15:16:11.414288 17533 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:42519 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.43.190:42519: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
I20260925 15:16:11.415241 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.407243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.8.43.190:42519)
0925 15:16:11.407365 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.407366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.407920 (+ 554us) server_negotiation.cc:407] Connection header received
0925 15:16:11.407993 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.407993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.408005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.408062 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.408649 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.409152 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.409935 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.410134 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.410222 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.411676 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:11.411680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:11.411680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:11.411690 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:11.413568 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:11.414109 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:11.414112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:11.414112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:11.414261 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:11.414495 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:11.414496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:11.414497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:11.414807 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:11.414941 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:11.415016 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:16:11.415120 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:11.418848 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.414529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:47745)
0925 15:16:11.414573 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.414581 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.414614 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.414752 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.414752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.414897 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.414950 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.414952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.415781 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.415781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.416458 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.416460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.416769 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.416770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.416816 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.417151 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.417155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.418774 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:16:11.418898 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.414566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:47745 (local address 127.8.43.190:42519)
0925 15:16:11.414599 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.414601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.414608 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:16:11.414663 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.414664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.414671 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.414717 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.415035 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.415655 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.416521 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.416694 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.416744 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.418841 (+ 2097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:47745: BlockingRecv error: recv got EOF from 127.8.43.131:47745 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:11 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:11.422881 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.419133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:38663)
0925 15:16:11.419164 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.419173 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.419255 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.419364 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.419365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.419482 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:11.419504 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.419505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.420103 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.420104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.420576 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.420577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.420934 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.420935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.420986 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.421359 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:11.421364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:11.422842 (+ 1478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 15:16:11.423004 17533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:16:11.423007 17270 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:11.419195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:38663 (local address 127.8.43.190:42519)
0925 15:16:11.419270 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.419272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.419280 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:16:11.419310 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.419311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.419319 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.419363 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.419578 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.420011 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.420669 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.420854 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.420933 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.422936 (+ 2003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:38663: BlockingRecv error: recv got EOF from 127.8.43.131:38663 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:16:12 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:12 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:12.166268 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:12.161629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:57441)
0925 15:16:12.161772 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:12.161783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:12.161838 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:12.161999 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:12.162000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:12.162170 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:12.162244 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:12.162246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.162898 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.162898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:12.163515 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:12.163518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.163873 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.163874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:12.163911 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:12.164435 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:12.164440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:12.166192 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":113,"threads_started":1}
W20260925 15:16:12.166355 17539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:12.161726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:57441 (local address 127.8.43.190:42519)
0925 15:16:12.161885 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:12.161886 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:12.161895 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:12.161915 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:12.161915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:12.161926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:12.161991 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:12.162294 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.162825 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.163598 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.163805 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.163867 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:12.166285 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:57441: BlockingRecv error: recv got EOF from 127.8.43.129:57441 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
Sep 25 15:16:12 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:12 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:12.299916 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:12.295482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:50881)
0925 15:16:12.295611 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:12.295621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:12.295680 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:12.295822 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:12.295823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:12.296018 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:12.296063 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:12.296065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.296667 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.296668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:12.297189 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:12.297191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.297515 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.297516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:12.297554 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:12.297974 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:12.297979 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:12.299856 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
W20260925 15:16:12.300033 17539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:12.295519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:50881 (local address 127.8.43.190:42519)
0925 15:16:12.295583 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:12.295585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:12.295678 (+ 93us) server_negotiation.cc:407] Connection header received
0925 15:16:12.295739 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:12.295740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:12.295753 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:12.295808 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:12.296122 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.296603 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.297263 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.297455 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.297506 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:12.299938 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:50881: BlockingRecv error: recv got EOF from 127.8.43.130:50881 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 15:16:12 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:12 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:12.428344 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:12.423470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:39899)
0925 15:16:12.423591 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:12.423604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:12.423674 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:12.423854 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:12.423855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:12.424032 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:12.424075 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:12.424077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.424759 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.424760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:12.425310 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:12.425312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.425685 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.425686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:12.425753 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:12.426258 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:12.426263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:12.428276 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
W20260925 15:16:12.428406 17539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:12.423569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:39899 (local address 127.8.43.190:42519)
0925 15:16:12.423616 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:12.423619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:12.423678 (+ 59us) server_negotiation.cc:407] Connection header received
0925 15:16:12.423760 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:12.423761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:12.423779 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:12.423847 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:12.424175 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.424669 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.425396 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:12.425594 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:12.425683 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:12.428358 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:39899: BlockingRecv error: recv got EOF from 127.8.43.131:39899 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 15:16:13 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:13 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:13.171363 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.166796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:45771)
0925 15:16:13.166941 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.166954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:13.167027 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:13.167153 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:13.167154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:13.167342 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:13.167392 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:13.167393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.168061 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.168062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:13.168604 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:13.168606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.168940 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.168941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:13.168977 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:13.169406 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:13.169410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:13.171212 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
W20260925 15:16:13.171411 17543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:13.166895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:45771 (local address 127.8.43.190:42519)
0925 15:16:13.167022 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:13.167024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:13.167032 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:16:13.167071 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:13.167072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:13.167081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:13.167136 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:13.167491 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.167986 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.168682 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.168878 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.168930 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:13.171333 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:45771: BlockingRecv error: recv got EOF from 127.8.43.129:45771 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
Sep 25 15:16:13 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:13 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:13.304677 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.300323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:59225)
0925 15:16:13.300465 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.300478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:13.300546 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:13.300718 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:13.300719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:13.300898 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:13.300947 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:13.300949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.301579 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.301580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:13.302098 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:13.302100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.302407 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.302408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:13.302446 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:13.302960 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:13.302972 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:13.304620 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":113,"threads_started":1}
W20260925 15:16:13.304828 17543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:13.300396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:59225 (local address 127.8.43.190:42519)
0925 15:16:13.300465 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:13.300468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:13.300544 (+ 76us) server_negotiation.cc:407] Connection header received
0925 15:16:13.300622 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:13.300623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:13.300636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:13.300690 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:13.301015 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.301512 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.302168 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.302346 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.302395 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:13.304711 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:59225: BlockingRecv error: recv got EOF from 127.8.43.130:59225 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 15:16:13 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:13 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:13.433694 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.428809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:37431)
0925 15:16:13.428939 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.428955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:13.429031 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:13.429180 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:13.429181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:13.429359 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:13.429404 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:13.429406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.430239 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.430240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:13.430793 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:13.430795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.431310 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.431311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:13.431355 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:13.431855 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:13.431860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:13.433627 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
W20260925 15:16:13.433763 17543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:13.428958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:37431 (local address 127.8.43.190:42519)
0925 15:16:13.429021 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:13.429024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:13.429032 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:16:13.429089 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:13.429090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:13.429102 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:13.429175 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:13.429487 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.430149 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.430869 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:13.431182 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:13.431292 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:13.433707 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:37431: BlockingRecv error: recv got EOF from 127.8.43.131:37431 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 15:16:14 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:14 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:14.176863 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:14.172005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:43175)
0925 15:16:14.172148 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:14.172176 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:14.172232 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:14.172381 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:14.172382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:14.172535 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:14.172581 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:14.172583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.173223 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.173224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:14.173988 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:14.173992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.174393 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.174394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:14.174449 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:14.174947 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:14.174952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:14.176757 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
W20260925 15:16:14.176908 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:14.172074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:43175 (local address 127.8.43.190:42519)
0925 15:16:14.172210 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:14.172212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:14.172221 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:14.172296 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:14.172297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:14.172307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:14.172366 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:14.172645 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.173136 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.174078 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.174317 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.174395 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:14.176837 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:43175: BlockingRecv error: recv got EOF from 127.8.43.129:43175 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":47,"threads_started":1}
Sep 25 15:16:14 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:14 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:14.310125 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:14.305232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:52503)
0925 15:16:14.305416 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:14.305427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:14.305510 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:14.305636 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:14.305636 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:14.305994 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:14.306042 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:14.306045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.306738 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.306740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:14.307310 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:14.307311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.307649 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.307650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:14.307695 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:14.308205 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:14.308211 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:14.310054 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
W20260925 15:16:14.310289 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:14.305294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:52503 (local address 127.8.43.190:42519)
0925 15:16:14.305371 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:14.305373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:14.305495 (+ 122us) server_negotiation.cc:407] Connection header received
0925 15:16:14.305565 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:14.305566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:14.305578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:14.305623 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:14.306147 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.306647 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.307391 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.307574 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.307650 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:14.310202 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:52503: BlockingRecv error: recv got EOF from 127.8.43.130:52503 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:16:14 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:14 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:14.438916 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:14.434196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:39309)
0925 15:16:14.434337 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:14.434348 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:14.434433 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:14.434586 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:14.434587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:14.434779 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:14.434825 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:14.434827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.435546 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.435548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:14.436082 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:14.436084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.436448 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.436449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:14.436509 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:14.437028 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:14.437032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:14.438823 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
W20260925 15:16:14.438963 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:14.434246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:39309 (local address 127.8.43.190:42519)
0925 15:16:14.434291 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:14.434294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:14.434414 (+ 120us) server_negotiation.cc:407] Connection header received
0925 15:16:14.434494 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:14.434495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:14.434508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:14.434579 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:14.434904 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.435469 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.436169 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:14.436363 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:14.436447 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:14.438898 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:39309: BlockingRecv error: recv got EOF from 127.8.43.131:39309 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 15:16:15 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:15 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:15.187511 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:15.177352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:48989)
0925 15:16:15.177526 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:15.177537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:15.177588 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:15.177756 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:15.177757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:15.177927 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:15.177973 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:15.177975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.178628 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.178629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:15.179176 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:15.179178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.179571 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.179572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:15.179613 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:15.185526 (+ 5913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:15.185531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:15.187444 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":112,"threads_started":1}
W20260925 15:16:15.187625 17551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:15.177448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:48989 (local address 127.8.43.190:42519)
0925 15:16:15.177573 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:15.177575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:15.177585 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:16:15.177633 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:15.177633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:15.177644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:15.177716 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:15.178057 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.178557 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.179312 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.179502 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.179561 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:15.187536 (+ 7975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:48989: BlockingRecv error: recv got EOF from 127.8.43.129:48989 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
Sep 25 15:16:15 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:15 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:15.315862 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:15.310607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:56251)
0925 15:16:15.310753 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:15.310770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:15.310833 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:15.310979 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:15.310980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:15.311144 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:15.311210 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:15.311212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.312261 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.312262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:15.313045 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:15.313048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.313497 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.313498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:15.313553 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:15.314049 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:15.314054 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:15.315804 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
W20260925 15:16:15.315984 17551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:15.310684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:56251 (local address 127.8.43.190:42519)
0925 15:16:15.310772 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:15.310774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:15.310830 (+ 56us) server_negotiation.cc:407] Connection header received
0925 15:16:15.310890 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:15.310891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:15.310901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:15.310956 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:15.311331 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.312053 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.313131 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.313428 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.313500 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:15.315889 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:56251: BlockingRecv error: recv got EOF from 127.8.43.130:56251 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 15:16:15 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:15 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:15.443987 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:15.439413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:46715)
0925 15:16:15.439559 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:15.439572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:15.439633 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:15.439771 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:15.439772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:15.439922 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:15.439969 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:15.439971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.440605 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.440605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:15.441139 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:15.441141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.441525 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.441525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:15.441587 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:15.442184 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:15.442197 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:15.443932 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
W20260925 15:16:15.444116 17551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:15.439518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:46715 (local address 127.8.43.190:42519)
0925 15:16:15.439562 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:15.439564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:15.439635 (+ 71us) server_negotiation.cc:407] Connection header received
0925 15:16:15.439679 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:15.439680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:15.439692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:15.439748 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:15.440027 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.440545 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.441223 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:15.441451 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:15.441522 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:15.444034 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:46715: BlockingRecv error: recv got EOF from 127.8.43.131:46715 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 15:16:16 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:16 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:16.192605 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:16.188019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:49425)
0925 15:16:16.188132 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:16.188140 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:16.188184 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:16.188521 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:16.188522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:16.188788 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:16.188849 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:16.188851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.189508 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.189509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:16.190041 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:16.190043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.190370 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.190371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:16.190408 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:16.190820 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:16.190825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:16.192547 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
W20260925 15:16:16.192756 17555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:16.188135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:49425 (local address 127.8.43.190:42519)
0925 15:16:16.188325 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:16.188327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:16.188336 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:16.188391 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:16.188392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:16.188406 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:16.188495 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:16.188934 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.189443 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.190113 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.190306 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.190358 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:16.192638 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:49425: BlockingRecv error: recv got EOF from 127.8.43.129:49425 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":239,"threads_started":1}
Sep 25 15:16:16 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:16 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:16.321146 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:16.316303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:49921)
0925 15:16:16.316458 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:16.316469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:16.316515 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:16.316699 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:16.316699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:16.316952 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:16.317038 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:16.317041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.317908 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.317908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:16.318429 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:16.318431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.318756 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.318757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:16.318794 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:16.319260 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:16.319264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:16.321043 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
W20260925 15:16:16.321208 17555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:16.316431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:49921 (local address 127.8.43.190:42519)
0925 15:16:16.316500 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:16.316502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:16.316510 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:16:16.316575 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:16.316576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:16.316587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:16.316694 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:16.317130 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.317837 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.318500 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.318695 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.318765 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:16.321133 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:49921: BlockingRecv error: recv got EOF from 127.8.43.130:49921 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 15:16:16 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:16 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:16.449467 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:16.444458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:52249)
0925 15:16:16.444619 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:16.444629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:16.444720 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:16.444950 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:16.444951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:16.445137 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:16.445182 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:16.445183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.445863 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.445864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:16.446393 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:16.446395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.446771 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.446772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:16.446846 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:16.447379 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:16.447390 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:16.449374 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
W20260925 15:16:16.449504 17555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:16.444539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:52249 (local address 127.8.43.190:42519)
0925 15:16:16.444611 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:16.444614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:16.444703 (+ 89us) server_negotiation.cc:407] Connection header received
0925 15:16:16.444855 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:16.444856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:16.444871 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:16.444944 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:16.445276 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.445781 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.446479 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:16.446679 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:16.446764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:16.449444 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:52249: BlockingRecv error: recv got EOF from 127.8.43.131:52249 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 15:16:17 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:17 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:17.197732 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:17.193039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:38743)
0925 15:16:17.193362 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:17.193395 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:17.193460 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:17.193587 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:17.193588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:17.193748 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:17.193819 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:17.193820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.194474 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.194475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:17.195005 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:17.195007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.195337 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.195337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:17.195404 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:17.195919 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:17.195924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:17.197677 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":229,"threads_started":1}
W20260925 15:16:17.197870 17559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:17.193208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:38743 (local address 127.8.43.190:42519)
0925 15:16:17.193385 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:17.193389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:17.193459 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:16:17.193503 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:17.193504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:17.193515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:17.193572 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:17.193883 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.194385 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.195070 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.195285 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.195335 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:17.197776 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:38743: BlockingRecv error: recv got EOF from 127.8.43.129:38743 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
Sep 25 15:16:17 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:17 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:17.326542 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:17.321784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:54557)
0925 15:16:17.321939 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:17.321950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:17.322013 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:17.322150 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:17.322151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:17.322326 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:17.322371 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:17.322373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.323035 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.323036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:17.323660 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:17.323663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.324016 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.324017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:17.324061 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:17.324579 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:17.324585 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:17.326481 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
W20260925 15:16:17.326666 17559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:17.321866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:54557 (local address 127.8.43.190:42519)
0925 15:16:17.321941 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:17.321944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:17.322014 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:16:17.322057 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:17.322058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:17.322070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:17.322134 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:17.322432 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.322966 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.323743 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.323949 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.324010 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:17.326588 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:54557: BlockingRecv error: recv got EOF from 127.8.43.130:54557 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 15:16:17 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:17 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:17.454447 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:17.449912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:36963)
0925 15:16:17.450067 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:17.450079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:17.450144 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:17.450282 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:17.450284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:17.450440 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:17.450486 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:17.450488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.451123 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.451124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:17.451724 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:17.451726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.452077 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.452078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:17.452115 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:17.452519 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:17.452523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:17.454355 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
W20260925 15:16:17.454485 17559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:17.449983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:36963 (local address 127.8.43.190:42519)
0925 15:16:17.450054 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:17.450056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:17.450143 (+ 87us) server_negotiation.cc:407] Connection header received
0925 15:16:17.450186 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:17.450187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:17.450197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:17.450247 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:17.450547 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.451052 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.451810 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:17.452014 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:17.452064 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:17.454426 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:36963: BlockingRecv error: recv got EOF from 127.8.43.131:36963 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:16:18 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:18 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:18.203171 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:18.198239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:55635)
0925 15:16:18.198431 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:18.198442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:18.198495 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:18.198656 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:18.198657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:18.198839 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:18.198889 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:18.198890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.199614 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.199614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:18.200124 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:18.200125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.200585 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.200586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:18.200641 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:18.201230 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:18.201236 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:18.203109 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":129,"threads_started":1}
W20260925 15:16:18.203353 17563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:18.198343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:55635 (local address 127.8.43.190:42519)
0925 15:16:18.198465 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:18.198467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:18.198483 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:16:18.198564 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:18.198565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:18.198577 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:18.198647 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:18.198961 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.199550 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.200216 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.200483 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.200547 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:18.203200 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:55635: BlockingRecv error: recv got EOF from 127.8.43.129:55635 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
Sep 25 15:16:18 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:18 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:18.331552 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:18.327038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:54249)
0925 15:16:18.327196 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:18.327277 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:18.327345 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:18.327509 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:18.327510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:18.327664 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:18.327717 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:18.327718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.328419 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.328420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:18.328971 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:18.328973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.329334 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.329335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:18.329372 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:18.329793 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:18.329797 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:18.331495 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
W20260925 15:16:18.331656 17563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:18.327106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:54249 (local address 127.8.43.190:42519)
0925 15:16:18.327176 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:18.327178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:18.327342 (+ 164us) server_negotiation.cc:407] Connection header received
0925 15:16:18.327393 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:18.327393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:18.327409 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:18.327461 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:18.327837 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.328340 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.329054 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.329256 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.329325 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:18.331580 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:54249: BlockingRecv error: recv got EOF from 127.8.43.130:54249 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 15:16:18 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:18 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:18.459105 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:18.454895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:43931)
0925 15:16:18.455033 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:18.455045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:18.455114 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:18.455247 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:18.455248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:18.455409 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:18.455456 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:18.455457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.456060 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.456061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:18.456588 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:18.456590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.456890 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.456890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:18.456928 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:18.457348 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:18.457353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:18.459046 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260925 15:16:18.459198 17563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:18.454997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:43931 (local address 127.8.43.190:42519)
0925 15:16:18.455067 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:18.455069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:18.455108 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:16:18.455162 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:18.455162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:18.455174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:18.455244 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:18.455514 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.455995 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.456654 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:18.456829 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:18.456877 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:18.459131 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:43931: BlockingRecv error: recv got EOF from 127.8.43.131:43931 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 15:16:19 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:19 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:19.208581 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:19.203654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:34887)
0925 15:16:19.203800 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:19.203811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:19.203868 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:19.204102 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:19.204103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:19.204254 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:19.204307 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:19.204308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.205019 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.205019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:19.205534 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:19.205535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.205919 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.205920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:19.205956 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:19.206427 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:19.206431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:19.208470 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
W20260925 15:16:19.208635 17567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:19.203823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:34887 (local address 127.8.43.190:42519)
0925 15:16:19.203975 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:19.203977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:19.203986 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:19.204008 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:19.204009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:19.204021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:19.204093 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:19.204381 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.204944 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.205618 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.205849 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.205915 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:19.208558 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:34887: BlockingRecv error: recv got EOF from 127.8.43.129:34887 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
Sep 25 15:16:19 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:19 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:19.336395 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:19.332029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:36849)
0925 15:16:19.332159 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:19.332170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:19.332232 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:19.332375 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:19.332375 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:19.332531 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:19.332578 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:19.332580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.333326 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.333326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:19.333839 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:19.333840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.334145 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.334146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:19.334182 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:19.334594 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:19.334598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:19.336298 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260925 15:16:19.336427 17567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:19.332100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:36849 (local address 127.8.43.190:42519)
0925 15:16:19.332159 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:19.332161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:19.332231 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:16:19.332286 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:19.332287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:19.332297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:19.332357 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:19.332638 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.333264 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.333908 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.334086 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.334135 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:19.336369 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:36849: BlockingRecv error: recv got EOF from 127.8.43.130:36849 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 15:16:19 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:19 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:19.464190 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:19.459612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:53129)
0925 15:16:19.459734 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:19.459747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:19.459804 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:19.459937 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:19.459938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:19.460093 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:19.460139 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:19.460141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.460851 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.460852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:19.461430 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:19.461432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.461811 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.461812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:19.461849 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:19.462371 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:19.462376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:19.464133 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
W20260925 15:16:19.464326 17567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:19.459693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:53129 (local address 127.8.43.190:42519)
0925 15:16:19.459770 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:19.459772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:19.459793 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:16:19.459864 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:19.459865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:19.459877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:19.459922 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:19.460204 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.460754 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.461513 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:19.461695 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:19.461816 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:19.464223 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:53129: BlockingRecv error: recv got EOF from 127.8.43.131:53129 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 15:16:20 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:20 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:20.214265 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:20.209073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:58035)
0925 15:16:20.209249 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:20.209260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:20.209314 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:20.209454 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:20.209454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:20.209618 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:20.209667 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:20.209668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.210349 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.210350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:20.211037 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:20.211040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.211596 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.211598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:20.211655 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:20.212239 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:20.212244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:20.214194 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
W20260925 15:16:20.214424 17571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:20.209149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:58035 (local address 127.8.43.190:42519)
0925 15:16:20.209292 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:20.209294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:20.209302 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:16:20.209361 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:20.209361 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:20.209372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:20.209433 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:20.209730 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.210230 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.211140 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.211473 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.211595 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:20.214322 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:58035: BlockingRecv error: recv got EOF from 127.8.43.129:58035 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
Sep 25 15:16:20 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:20 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:20.350915 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:20.336881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:34001)
0925 15:16:20.337012 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:20.337023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:20.337088 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:20.337217 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:20.337217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:20.337385 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:20.337428 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:20.337430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.338100 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.338100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:20.338616 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:20.338617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.338951 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.338952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:20.338993 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:20.349040 (+ 10047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:20.349045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:20.350858 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
W20260925 15:16:20.351034 17571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:20.336963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:34001 (local address 127.8.43.190:42519)
0925 15:16:20.337056 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:20.337058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:20.337072 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:16:20.337149 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:20.337150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:20.337163 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:20.337212 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:20.337531 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.338026 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.338691 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.338886 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.338941 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:20.350955 (+ 12014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:34001: BlockingRecv error: recv got EOF from 127.8.43.130:34001 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:16:20 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:20 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:20.469516 17573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:20.464672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:42979)
0925 15:16:20.464879 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:20.464890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:20.464950 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:20.465111 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:20.465112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:20.465279 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:20.465327 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:20.465328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.465998 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.465999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:20.466510 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:20.466512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.466868 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.466869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:20.466910 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:20.467436 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:20.467440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:20.469424 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":145,"threads_started":1}
W20260925 15:16:20.469547 17571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:20.464728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:42979 (local address 127.8.43.190:42519)
0925 15:16:20.464799 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:20.464801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:20.464954 (+ 153us) server_negotiation.cc:407] Connection header received
0925 15:16:20.464994 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:20.464995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:20.465009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:20.465065 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:20.465399 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.465924 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.466593 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:20.466794 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:20.466857 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:20.469489 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:42979: BlockingRecv error: recv got EOF from 127.8.43.131:42979 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 15:16:21 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:21 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:21.219444 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:21.214737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:50849)
0925 15:16:21.214909 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:21.214933 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:21.214988 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:21.215148 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:21.215149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:21.215344 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:21.215394 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:21.215396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.216053 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.216053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:21.216590 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:21.216592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.216954 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.216954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:21.216996 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:21.217443 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:21.217447 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:21.219386 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
W20260925 15:16:21.219560 17575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:21.214824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:50849 (local address 127.8.43.190:42519)
0925 15:16:21.214974 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:21.214975 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:21.214984 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:21.215046 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:21.215047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:21.215056 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:21.215111 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:21.215460 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.215976 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.216663 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.216884 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.216936 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:21.219480 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:50849: BlockingRecv error: recv got EOF from 127.8.43.129:50849 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
Sep 25 15:16:21 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:21 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:21.356062 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:21.351361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:45497)
0925 15:16:21.351499 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:21.351510 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:21.351569 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:21.351697 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:21.351698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:21.351855 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:21.351902 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:21.351903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.352524 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.352526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:21.353176 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:21.353178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.353500 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.353501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:21.353538 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:21.354044 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:21.354049 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:21.355968 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
W20260925 15:16:21.356109 17575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:21.351430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:45497 (local address 127.8.43.190:42519)
0925 15:16:21.351506 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:21.351508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:21.351570 (+ 62us) server_negotiation.cc:407] Connection header received
0925 15:16:21.351626 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:21.351627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:21.351637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:21.351681 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:21.351962 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.352464 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.353248 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.353439 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.353491 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:21.356046 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:45497: BlockingRecv error: recv got EOF from 127.8.43.130:45497 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 15:16:21 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:21 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:21.474316 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:21.470019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:46665)
0925 15:16:21.470212 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:21.470223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:21.470298 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:21.470412 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:21.470413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:21.470568 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:21.470615 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:21.470617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.471263 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.471264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:21.471819 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:21.471821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.472126 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.472127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:21.472166 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:21.472595 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:21.472599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:21.474251 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
W20260925 15:16:21.474412 17575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:21.470132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:46665 (local address 127.8.43.190:42519)
0925 15:16:21.470201 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:21.470204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:21.470281 (+ 77us) server_negotiation.cc:407] Connection header received
0925 15:16:21.470347 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:21.470347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:21.470359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:21.470399 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:21.470672 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.471151 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.471891 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:21.472063 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:21.472111 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:21.474346 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:46665: BlockingRecv error: recv got EOF from 127.8.43.131:46665 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 15:16:22 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:22 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:22.224920 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:22.219963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:37741)
0925 15:16:22.220143 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:22.220155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:22.220215 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:22.220359 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:22.220360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:22.220591 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:22.220640 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:22.220642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.221308 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.221309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:22.221872 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:22.221874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.222305 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.222306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:22.222358 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:22.222892 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:22.222897 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:22.224807 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
W20260925 15:16:22.225010 17579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:22.220088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:37741 (local address 127.8.43.190:42519)
0925 15:16:22.220207 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:22.220209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:22.220218 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:22.220267 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:22.220267 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:22.220277 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:22.220339 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:22.220707 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.221219 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.221978 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.222224 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.222293 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:22.224926 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:37741: BlockingRecv error: recv got EOF from 127.8.43.129:37741 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
Sep 25 15:16:22 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:22 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:22.361014 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:22.356525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:45833)
0925 15:16:22.356652 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:22.356662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:22.356731 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:22.356851 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:22.356852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:22.357015 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:22.357058 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:22.357060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.357682 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.357682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:22.358203 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:22.358205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.358536 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.358537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:22.358576 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:22.359094 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:22.359099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:22.360904 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
W20260925 15:16:22.361042 17579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:22.356610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:45833 (local address 127.8.43.190:42519)
0925 15:16:22.356673 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:22.356676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:22.356700 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:16:22.356791 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:22.356791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:22.356801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:22.356837 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:22.357116 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.357616 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.358283 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.358469 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.358521 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:22.360974 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:45833: BlockingRecv error: recv got EOF from 127.8.43.130:45833 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 15:16:22 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:22 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:22.479414 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:22.474813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:41543)
0925 15:16:22.474972 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:22.474983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:22.475041 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:22.475204 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:22.475205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:22.475389 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:22.475437 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:22.475438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.476088 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.476089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:22.476624 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:22.476625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.476930 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.476931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:22.476970 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:22.477496 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:22.477501 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:22.479344 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
W20260925 15:16:22.479501 17579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:22.474884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:41543 (local address 127.8.43.190:42519)
0925 15:16:22.474935 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:22.474937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:22.475042 (+ 105us) server_negotiation.cc:407] Connection header received
0925 15:16:22.475086 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:22.475086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:22.475098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:22.475148 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:22.475517 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.476014 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.476694 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:22.476870 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:22.476918 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:22.479428 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:41543: BlockingRecv error: recv got EOF from 127.8.43.131:41543 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 15:16:23 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:23 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:23.230002 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:23.225448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:56343)
0925 15:16:23.225610 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:23.225625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:23.225669 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:23.225861 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:23.225861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:23.226026 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:23.226073 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:23.226074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.226706 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.226707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:23.227259 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:23.227261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.227618 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.227619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:23.227656 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:23.228185 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:23.228190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:23.229932 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260925 15:16:23.230157 17583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:23.225577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:56343 (local address 127.8.43.190:42519)
0925 15:16:23.225715 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:23.225717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:23.225724 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:16:23.225751 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:23.225751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:23.225760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:23.225841 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:23.226139 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.226638 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.227353 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.227549 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.227605 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:23.230055 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:56343: BlockingRecv error: recv got EOF from 127.8.43.129:56343 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
Sep 25 15:16:23 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:23 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:23.365986 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:23.361445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:52755)
0925 15:16:23.361607 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:23.361618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:23.361669 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:23.361811 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:23.361813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:23.361999 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:23.362056 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:23.362058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.362670 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.362671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:23.363229 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:23.363230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.363627 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.363628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:23.363680 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:23.364097 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:23.364101 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:23.365920 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
W20260925 15:16:23.366077 17583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:23.361523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:52755 (local address 127.8.43.190:42519)
0925 15:16:23.361581 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:23.361583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:23.361672 (+ 89us) server_negotiation.cc:407] Connection header received
0925 15:16:23.361718 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:23.361718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:23.361729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:23.361778 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:23.362116 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.362599 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.363369 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.363559 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.363611 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:23.366013 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:52755: BlockingRecv error: recv got EOF from 127.8.43.130:52755 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 15:16:23 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:23 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:23.485309 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:23.479984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:45865)
0925 15:16:23.480176 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:23.480188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:23.480273 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:23.480414 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:23.480415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:23.480604 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:23.480644 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:23.480646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.481498 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.481500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:23.482359 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:23.482361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.482719 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.482720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:23.482759 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:23.483184 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:23.483188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:23.485224 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":113,"threads_started":1}
W20260925 15:16:23.485420 17583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:23.480095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:45865 (local address 127.8.43.190:42519)
0925 15:16:23.480167 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:23.480170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:23.480255 (+ 85us) server_negotiation.cc:407] Connection header received
0925 15:16:23.480343 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:23.480344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:23.480358 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:23.480411 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:23.480726 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.481410 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.482446 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:23.482647 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:23.482706 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:23.485323 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:45865: BlockingRecv error: recv got EOF from 127.8.43.131:45865 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 15:16:24 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:24 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:24.235280 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:24.230512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:44967)
0925 15:16:24.230697 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:24.230707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:24.230769 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:24.230914 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:24.230915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:24.231102 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:24.231147 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:24.231149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.231943 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.231944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:24.232488 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:24.232490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.232885 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.232894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:24.232956 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:24.233393 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:24.233397 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:24.235141 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":110,"threads_started":1}
W20260925 15:16:24.235337 17587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:24.230550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:44967 (local address 127.8.43.190:42519)
0925 15:16:24.230682 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:24.230684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:24.230752 (+ 68us) server_negotiation.cc:407] Connection header received
0925 15:16:24.230832 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:24.230833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:24.230847 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:24.230900 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:24.231247 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.231862 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.232583 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.232800 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.232883 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:24.235243 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:44967: BlockingRecv error: recv got EOF from 127.8.43.129:44967 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
Sep 25 15:16:24 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:24 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:24.371258 17588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:24.366503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:53767)
0925 15:16:24.366681 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:24.366696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:24.366771 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:24.366887 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:24.366888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:24.367051 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:24.367095 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:24.367097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.367751 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.367753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:24.368303 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:24.368305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.368626 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.368627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:24.368679 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:24.369179 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:24.369184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:24.371158 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
W20260925 15:16:24.371373 17587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:24.366648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:53767 (local address 127.8.43.190:42519)
0925 15:16:24.366714 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:24.366716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:24.366746 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:16:24.366823 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:24.366824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:24.366834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:24.366882 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:24.367182 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.367679 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.368377 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.368558 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.368610 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:24.371306 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:53767: BlockingRecv error: recv got EOF from 127.8.43.130:53767 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 15:16:24 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:24 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:24.490832 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:24.485737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:44725)
0925 15:16:24.485877 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:24.485888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:24.485937 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:24.486071 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:24.486072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:24.486258 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:24.486306 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:24.486307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.486987 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.486988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:24.487585 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:24.487588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.487898 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.487899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:24.487938 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:24.488390 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:24.488395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:24.490767 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260925 15:16:24.490954 17587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:24.485825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:44725 (local address 127.8.43.190:42519)
0925 15:16:24.485915 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:24.485918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:24.485938 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:16:24.485981 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:24.485982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:24.485996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:24.486059 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:24.486403 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.486911 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.487657 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:24.487838 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:24.487886 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:24.490865 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:44725: BlockingRecv error: recv got EOF from 127.8.43.131:44725 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 15:16:25 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:25 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:25.240873 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:25.235912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:59357)
0925 15:16:25.236083 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:25.236104 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:25.236162 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:25.236386 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:25.236387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:25.236604 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:25.236651 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:25.236653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.237349 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.237350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:25.237886 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:25.237889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.238207 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.238208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:25.238243 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:25.238728 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:25.238733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:25.240765 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
W20260925 15:16:25.240911 17591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:25.235972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:59357 (local address 127.8.43.190:42519)
0925 15:16:25.236135 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:25.236137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:25.236148 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:16:25.236222 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:25.236222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:25.236236 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:25.236296 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:25.236741 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.237260 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.237961 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.238145 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.238202 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:25.240844 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:59357: BlockingRecv error: recv got EOF from 127.8.43.129:59357 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
Sep 25 15:16:25 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:25 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:25.381420 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:25.371683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:48663)
0925 15:16:25.371833 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:25.371844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:25.371903 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:25.372043 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:25.372043 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:25.372204 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:25.372249 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:25.372251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.372867 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.372868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:25.373413 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:25.373414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.373762 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.373763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:25.373798 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:25.379293 (+ 5495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:25.379298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:25.381320 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
W20260925 15:16:25.381531 17591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:25.371731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:48663 (local address 127.8.43.190:42519)
0925 15:16:25.371791 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:25.371793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:25.371902 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:16:25.371963 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:25.371964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:25.371975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:25.372024 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:25.372315 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.372797 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.373505 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.373695 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.373759 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:25.381457 (+ 7698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:48663: BlockingRecv error: recv got EOF from 127.8.43.130:48663 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 15:16:25 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:25 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.131@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:25.496217 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:25.491378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.131:50291)
0925 15:16:25.491499 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:25.491518 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:25.491575 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:25.491741 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:25.491743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:25.491914 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:25.491958 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:25.491960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.492627 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.492628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:25.493186 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:25.493188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.493504 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.493505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:25.493544 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:25.494051 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:25.494056 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:25.496121 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
W20260925 15:16:25.496253 17591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:25.491467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:50291 (local address 127.8.43.190:42519)
0925 15:16:25.491531 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:25.491533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:25.491566 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:16:25.491622 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:25.491623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:25.491635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:25.491688 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:25.492060 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.492552 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.493249 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:25.493438 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:25.493494 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:25.496193 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.131:50291: BlockingRecv error: recv got EOF from 127.8.43.131:50291 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 15:16:26 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:26 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.129@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:26.246121 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.241431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.129:41781)
0925 15:16:26.241549 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:26.241560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:26.241618 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:26.241752 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:26.241752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:26.241919 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:26.241969 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.241970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.242749 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.242750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.243354 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.243356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.243659 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.243660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.243696 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.244141 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:26.244146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:26.246052 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260925 15:16:26.246338 17595 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:26.241490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:41781 (local address 127.8.43.190:42519)
0925 15:16:26.241633 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:26.241635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:26.241644 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:26.241676 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:26.241676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:26.241688 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:26.241746 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:26.242033 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.242675 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.243425 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.243602 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.243650 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.246265 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.129:41781: BlockingRecv error: recv got EOF from 127.8.43.129:41781 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
Sep 25 15:16:26 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:16:26 dist-test-slave-c5d6 krb5kdc[16536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.43.130@KRBTEST.COM for oryx/127.8.43.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:16:26.386920 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.382000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:42519 (local address 127.8.43.130:34767)
0925 15:16:26.382126 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:26.382137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:26.382204 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:26.382389 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:26.382390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:26.382661 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:26.382713 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.382716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.383513 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.383514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.384126 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.384128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.384466 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.384467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.384508 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.384957 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:26.384962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:26.386864 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.43.190:42519: Server oryx/127.8.43.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
W20260925 15:16:26.387013 17595 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:26.382080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:34767 (local address 127.8.43.190:42519)
0925 15:16:26.382125 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:26.382129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:26.382189 (+ 60us) server_negotiation.cc:407] Connection header received
0925 15:16:26.382268 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:26.382269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:26.382282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:26.382346 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:26.382869 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.383419 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.384202 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.384398 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.384452 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.386948 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.43.130:34767: BlockingRecv error: recv got EOF from 127.8.43.130:34767 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:16:26.407330 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 17119
I20260925 15:16:26.412179 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 17271
I20260925 15:16:26.416149 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 17403
I20260925 15:16:26.420090 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 17049
2026-09-25T15:16:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17476 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T15:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:26Z Disabled control of system clock
I20260925 15:16:26.447077 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44239
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:16:26.533128 17600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:26.533339 17600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:26.533360 17600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:26.534598 17600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:26.534641 17600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:26.534654 17600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:26.534667 17600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:26.534678 17600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:26.536145 17600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:26.536417 17600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:26.536727 17600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:26.542018 17608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:26.541960 17606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:26.541963 17605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:26.541980 17600 server_base.cc:1034] running on GCE node
I20260925 15:16:26.542356 17600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:26.542672 17600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:26.543840 17600 hybrid_clock.cc:648] HybridClock initialized: now 1790349386543819 us; error 42 us; skew 500 ppm
I20260925 15:16:26.545125 17600 webserver.cc:460] Webserver started at http://127.8.43.190:44105/ using document root <none> and password file <none>
I20260925 15:16:26.545379 17600 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:26.545460 17600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:26.545599 17600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:26.546597 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "bbadb671ef66487c82dff25c474b3074"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "f66c1037406ca01956de3422c77f4b6a"
server_key_iv: "e13d87ceb158b19214af6ab777517382"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.546931 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "bbadb671ef66487c82dff25c474b3074"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "f66c1037406ca01956de3422c77f4b6a"
server_key_iv: "e13d87ceb158b19214af6ab777517382"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.548170 17600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:16:26.548869 17614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.549015 17600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:16:26.549113 17600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "bbadb671ef66487c82dff25c474b3074"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "f66c1037406ca01956de3422c77f4b6a"
server_key_iv: "e13d87ceb158b19214af6ab777517382"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.549181 17600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:26.558938 17600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:26.559175 17600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:26.559353 17600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:26.562876 17600 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44239
I20260925 15:16:26.562916 17666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44239 every 8 connection(s)
I20260925 15:16:26.563279 17600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:26.563944 17667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:26.566439 17667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074: Bootstrap starting.
I20260925 15:16:26.566996 17667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:26.567348 17667 log.cc:824] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:26.567983 17667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074: No bootstrap required, opened a new log
I20260925 15:16:26.569469 17667 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbadb671ef66487c82dff25c474b3074" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44239 } }
I20260925 15:16:26.569597 17667 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:26.569646 17667 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbadb671ef66487c82dff25c474b3074, State: Initialized, Role: FOLLOWER
I20260925 15:16:26.569787 17667 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbadb671ef66487c82dff25c474b3074" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44239 } }
I20260925 15:16:26.569849 17667 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:26.569869 17667 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:26.569933 17667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:26.570438 17667 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbadb671ef66487c82dff25c474b3074" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44239 } }
I20260925 15:16:26.570541 17667 leader_election.cc:302] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbadb671ef66487c82dff25c474b3074; no voters:
I20260925 15:16:26.570745 17667 leader_election.cc:288] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:16:26.570776 17670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:26.570930 17670 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [term 1 LEADER]: Becoming Leader. State: Replica: bbadb671ef66487c82dff25c474b3074, State: Running, Role: LEADER
I20260925 15:16:26.570999 17667 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:26.571033 17670 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbadb671ef66487c82dff25c474b3074" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44239 } }
I20260925 15:16:26.571470 17672 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbadb671ef66487c82dff25c474b3074. Latest consensus state: current_term: 1 leader_uuid: "bbadb671ef66487c82dff25c474b3074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbadb671ef66487c82dff25c474b3074" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44239 } } }
I20260925 15:16:26.571641 17672 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:26.571825 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 17600
I20260925 15:16:26.571925 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:16:26.572126 8366 external_mini_cluster.cc:1428] Setting key dc463a1d6a468a337cf41e08ed556140
I20260925 15:16:26.572184 17679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:26.571604 17671 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbadb671ef66487c82dff25c474b3074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbadb671ef66487c82dff25c474b3074" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44239 } } }
I20260925 15:16:26.572242 17671 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:26.572654 17679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:26.574182 17679 catalog_manager.cc:1347] Generated new cluster ID: 42f8f449837e4534987fce8fc500ec93
I20260925 15:16:26.574232 17679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:26.578018 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.572621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.8.43.190:44239)
0925 15:16:26.572807 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:26.572857 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:26.573431 (+ 574us) server_negotiation.cc:407] Connection header received
0925 15:16:26.573548 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:26.573555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:26.573622 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:26.573775 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:26.573989 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.574955 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.576015 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.576454 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.576597 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.577295 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:26.577299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:26.577300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:16:26.577475 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:26.577519 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:16:26.577529 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:26.577604 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:26.577764 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:16:26.577846 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260925 15:16:26.590997 17679 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:26.591559 17679 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:26.596084 17679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbadb671ef66487c82dff25c474b3074: Generated new TSK 0
I20260925 15:16:26.596251 17679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:16:26.601464 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44239
--builtin_ntp_servers=127.8.43.148:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 15:16:26.680701 17691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:26.680908 17691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:26.680929 17691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:26.682049 17691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:26.682089 17691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:26.682112 17691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:16:26.683442 17691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:26.683679 17691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:26.683991 17691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:26.689637 17696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:26.689664 17697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:26.689900 17691 server_base.cc:1034] running on GCE node
W20260925 15:16:26.689937 17699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:26.690126 17691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:26.690375 17691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:26.691538 17691 hybrid_clock.cc:648] HybridClock initialized: now 1790349386691527 us; error 31 us; skew 500 ppm
I20260925 15:16:26.692765 17691 webserver.cc:460] Webserver started at http://127.8.43.129:32777/ using document root <none> and password file <none>
I20260925 15:16:26.693009 17691 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:26.693079 17691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:26.693212 17691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:26.694213 17691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "c71dbfd6e19b46158eea840b622a6b09"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "f9cd28f838880eada3eedd492172dba5"
server_key_iv: "4b34466ce217f1ef394d06707795c987"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.694602 17691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "c71dbfd6e19b46158eea840b622a6b09"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "f9cd28f838880eada3eedd492172dba5"
server_key_iv: "4b34466ce217f1ef394d06707795c987"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.696089 17691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:16:26.696902 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.697069 17691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:26.697162 17691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "c71dbfd6e19b46158eea840b622a6b09"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "f9cd28f838880eada3eedd492172dba5"
server_key_iv: "4b34466ce217f1ef394d06707795c987"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.697250 17691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:26.708635 17691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:26.708915 17691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:26.709059 17691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:26.709270 17691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:26.709569 17691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:26.709621 17691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.709661 17691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:26.709687 17691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.715391 17691 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:44089
I20260925 15:16:26.715478 17818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:44089 every 8 connection(s)
I20260925 15:16:26.715778 17691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:26.716372 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 17691
I20260925 15:16:26.716461 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:16:26.716684 8366 external_mini_cluster.cc:1428] Setting key d3e702d212a2248789c4f7630b58f18f
I20260925 15:16:26.717659 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44239
--builtin_ntp_servers=127.8.43.148:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:16:26.722110 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.716519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:46403 (local address 127.8.43.190:44239)
0925 15:16:26.716565 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:26.716567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:26.716808 (+ 241us) server_negotiation.cc:407] Connection header received
0925 15:16:26.717420 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:26.717421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:26.717434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:26.717518 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:26.718209 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.718979 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.720493 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.720690 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.720742 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.721851 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:26.721852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:26.721852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:16:26.721856 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:26.721869 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:16:26.721873 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:26.721920 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:26.722031 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:16:26.722054 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:16:26.722429 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.716509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44239 (local address 127.8.43.129:46403)
0925 15:16:26.716682 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:26.716714 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:26.717339 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:26.717570 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:26.717573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:26.717581 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:26.718123 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.718128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.719080 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.719081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.720417 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.720420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.720770 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.720771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.720929 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.721669 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:16:26.721669 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:26.721712 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:16:26.721743 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:26.721943 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:26.721949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:26.721952 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:16:26.722031 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:16:26.722155 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260925 15:16:26.722958 17819 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44239
I20260925 15:16:26.723063 17819 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:26.723299 17819 heartbeater.cc:511] Master 127.8.43.190:44239 requested a full tablet report, sending...
I20260925 15:16:26.723845 17631 ts_manager.cc:194] Registered new tserver with Master: c71dbfd6e19b46158eea840b622a6b09 (127.8.43.129:44089)
I20260925 15:16:26.724443 17631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.129:46403
W20260925 15:16:26.799932 17822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:26.800122 17822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:26.800143 17822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:26.801318 17822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:26.801357 17822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:26.801381 17822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:16:26.802742 17822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:26.802984 17822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:26.803310 17822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:26.808094 17830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:26.808054 17827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:26.808115 17828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:26.808346 17822 server_base.cc:1034] running on GCE node
I20260925 15:16:26.808537 17822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:26.808717 17822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:26.809850 17822 hybrid_clock.cc:648] HybridClock initialized: now 1790349386809836 us; error 29 us; skew 500 ppm
I20260925 15:16:26.810926 17822 webserver.cc:460] Webserver started at http://127.8.43.130:40841/ using document root <none> and password file <none>
I20260925 15:16:26.811158 17822 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:26.811201 17822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:26.811391 17822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:26.812407 17822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "dcf892167c3646e08f202f32d344c823"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "1100b39776c71ec52b30c3dde8d588ab"
server_key_iv: "e4c3bbcd3d6c0808a00f5ac59817e470"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.812769 17822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "dcf892167c3646e08f202f32d344c823"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "1100b39776c71ec52b30c3dde8d588ab"
server_key_iv: "e4c3bbcd3d6c0808a00f5ac59817e470"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.814038 17822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:16:26.814706 17836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.814857 17822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:26.814940 17822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "dcf892167c3646e08f202f32d344c823"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "1100b39776c71ec52b30c3dde8d588ab"
server_key_iv: "e4c3bbcd3d6c0808a00f5ac59817e470"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.814992 17822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:26.824800 17822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:26.825029 17822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:26.825126 17822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:26.825279 17822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:26.825604 17822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:26.825637 17822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.825676 17822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:26.825723 17822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.831319 17822 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:41807
I20260925 15:16:26.831380 17949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:41807 every 8 connection(s)
I20260925 15:16:26.831688 17822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:26.832995 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 17822
I20260925 15:16:26.833083 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:16:26.833276 8366 external_mini_cluster.cc:1428] Setting key 3b2a99bd5ced34ef011ae9f7c2ffa281
I20260925 15:16:26.834283 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44239
--builtin_ntp_servers=127.8.43.148:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:16:26.837641 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.832427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:57583 (local address 127.8.43.190:44239)
0925 15:16:26.832490 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:26.832492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:26.832732 (+ 240us) server_negotiation.cc:407] Connection header received
0925 15:16:26.833556 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:26.833557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:26.833569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:26.833618 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:26.834430 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.834970 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.836193 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.836392 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.836466 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.837341 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:26.837342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:26.837343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:16:26.837347 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:26.837361 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:16:26.837365 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:26.837433 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:26.837541 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:16:26.837566 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:16:26.837858 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.832417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44239 (local address 127.8.43.130:57583)
0925 15:16:26.832610 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:26.832642 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:26.833418 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:26.833692 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:26.833695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:26.833703 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:26.834306 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.834322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.835061 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.835062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.836072 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.836075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.836485 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.836486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.836612 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.837165 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:16:26.837166 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:26.837208 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:16:26.837243 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:26.837433 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:26.837440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:26.837443 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:16:26.837541 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:16:26.837641 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260925 15:16:26.838291 17950 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44239
I20260925 15:16:26.838377 17950 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:26.838569 17950 heartbeater.cc:511] Master 127.8.43.190:44239 requested a full tablet report, sending...
I20260925 15:16:26.838971 17631 ts_manager.cc:194] Registered new tserver with Master: dcf892167c3646e08f202f32d344c823 (127.8.43.130:41807)
I20260925 15:16:26.839557 17631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.130:57583
W20260925 15:16:26.914295 17953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:26.914482 17953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:26.914502 17953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:26.915578 17953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:26.915612 17953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:26.915637 17953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:16:26.916846 17953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:26.917025 17953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:26.917310 17953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:26.922200 17961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:26.922266 17958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:26.922355 17959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:26.922457 17953 server_base.cc:1034] running on GCE node
I20260925 15:16:26.922854 17953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:26.923090 17953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:26.924247 17953 hybrid_clock.cc:648] HybridClock initialized: now 1790349386924229 us; error 36 us; skew 500 ppm
I20260925 15:16:26.925468 17953 webserver.cc:460] Webserver started at http://127.8.43.131:44727/ using document root <none> and password file <none>
I20260925 15:16:26.925721 17953 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:26.925792 17953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:26.925933 17953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:26.926944 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "d7785a9ce9484102a956ad56c5cffd21"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "04906e34f025602b4b57746aeea2e5e8"
server_key_iv: "f806268c627b939123682c49f769b152"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.927366 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "d7785a9ce9484102a956ad56c5cffd21"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "04906e34f025602b4b57746aeea2e5e8"
server_key_iv: "f806268c627b939123682c49f769b152"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.928639 17953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:16:26.929311 17967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.929472 17953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:16:26.929565 17953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "d7785a9ce9484102a956ad56c5cffd21"
format_stamp: "Formatted at 2026-09-25 15:16:26 on dist-test-slave-c5d6"
server_key: "04906e34f025602b4b57746aeea2e5e8"
server_key_iv: "f806268c627b939123682c49f769b152"
server_key_version: "encryptionkey@0"
I20260925 15:16:26.929651 17953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:26.938153 17953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:26.938411 17953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:26.938552 17953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:26.938753 17953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:26.939051 17953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:26.939105 17953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.939142 17953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:26.939168 17953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:26.944810 17953 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:42967
I20260925 15:16:26.944868 18080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:42967 every 8 connection(s)
I20260925 15:16:26.945155 17953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:26.948933 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 17953
I20260925 15:16:26.949040 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:16:26.949270 8366 external_mini_cluster.cc:1428] Setting key 2eba441eda0f4a01617d5e40c488cfc2
I20260925 15:16:26.950328 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.945801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:45983 (local address 127.8.43.190:44239)
0925 15:16:26.945880 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:26.945883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:26.946056 (+ 173us) server_negotiation.cc:407] Connection header received
0925 15:16:26.946499 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:26.946500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:26.946510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:26.946560 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:26.947209 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.947802 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.948737 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.949044 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.949125 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.950001 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:26.950003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:26.950003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:16:26.950008 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:26.950024 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:16:26.950029 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:26.950092 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:26.950189 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 15:16:26.950220 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:16:26.950546 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.945759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44239 (local address 127.8.43.131:45983)
0925 15:16:26.945943 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:26.945974 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:26.946416 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:26.946608 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:26.946611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:26.946618 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:26.947094 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.947097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.947887 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.947888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.948648 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:26.948650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:26.949149 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:26.949150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:26.949253 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:26.949848 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:16:26.949848 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:26.949884 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:16:26.949910 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:26.950096 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:26.950100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:26.950102 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:16:26.950183 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:16:26.950300 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":103,"threads_started":1}
I20260925 15:16:26.951020 18081 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44239
I20260925 15:16:26.951122 18081 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:26.951367 18081 heartbeater.cc:511] Master 127.8.43.190:44239 requested a full tablet report, sending...
I20260925 15:16:26.951776 17631 ts_manager.cc:194] Registered new tserver with Master: d7785a9ce9484102a956ad56c5cffd21 (127.8.43.131:42967)
I20260925 15:16:26.952128 17631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.131:45983
I20260925 15:16:26.961038 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:16:26.962846 17689 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:26.962354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.8.43.190:44239)
0925 15:16:26.962436 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:26.962438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:26.962527 (+ 89us) server_negotiation.cc:407] Connection header received
0925 15:16:26.962558 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:26.962558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:26.962568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:26.962616 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:26.962744 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45708: BlockingRecv error: recv got EOF from 127.0.0.1:45708 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:16:26.962862 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 17691
I20260925 15:16:26.967824 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 17822
I20260925 15:16:26.972024 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 17953
I20260925 15:16:26.976122 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 17600
2026-09-25T15:16:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (555 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T15:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:26Z Disabled control of system clock
I20260925 15:16:27.002383 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38411
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:16:27.081413 18092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:27.081606 18092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:27.081627 18092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:27.082693 18092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:27.082726 18092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:27.082738 18092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:27.082751 18092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:27.082762 18092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:27.084021 18092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:38411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:27.084201 18092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:27.084427 18092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:27.089093 18100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:27.089140 18098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:27.089160 18097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:27.089300 18092 server_base.cc:1034] running on GCE node
I20260925 15:16:27.089527 18092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:27.089805 18092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:27.090992 18092 hybrid_clock.cc:648] HybridClock initialized: now 1790349387090988 us; error 37 us; skew 500 ppm
I20260925 15:16:27.092238 18092 webserver.cc:460] Webserver started at http://127.8.43.190:33141/ using document root <none> and password file <none>
I20260925 15:16:27.092474 18092 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:27.092542 18092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:27.092700 18092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:27.093698 18092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "3a0f43360d8540ff8d39cae4991da801"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "e67f8388d8fea967e24972a7139bdaae"
server_key_iv: "332083e26bafbee41cea5ef16de71cde"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.094079 18092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "3a0f43360d8540ff8d39cae4991da801"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "e67f8388d8fea967e24972a7139bdaae"
server_key_iv: "332083e26bafbee41cea5ef16de71cde"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.095474 18092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:16:27.096176 18106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.096342 18092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.096443 18092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "3a0f43360d8540ff8d39cae4991da801"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "e67f8388d8fea967e24972a7139bdaae"
server_key_iv: "332083e26bafbee41cea5ef16de71cde"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.096515 18092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:27.099793 18092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:27.100039 18092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:27.100179 18092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:27.103914 18092 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:38411
I20260925 15:16:27.103941 18158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:38411 every 8 connection(s)
I20260925 15:16:27.104246 18092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:27.104851 18159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:27.106220 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 18092
I20260925 15:16:27.106328 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:16:27.106498 8366 external_mini_cluster.cc:1428] Setting key cc55a9a2f2d4834dc863588d39b1f084
I20260925 15:16:27.107925 18159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801: Bootstrap starting.
I20260925 15:16:27.108572 18159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:27.108897 18159 log.cc:824] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:27.109046 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.107003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.8.43.190:38411)
0925 15:16:27.107195 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:27.107204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:27.107254 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:16:27.107746 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:27.107754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:27.107805 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:27.107950 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:27.108591 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:27.108594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:27.108594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:16:27.108599 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:27.108642 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:16:27.108656 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:27.108758 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:27.108788 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 15:16:27.108843 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260925 15:16:27.109593 18159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801: No bootstrap required, opened a new log
I20260925 15:16:27.110803 18159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0f43360d8540ff8d39cae4991da801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38411 } }
I20260925 15:16:27.111003 18159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:27.111063 18159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0f43360d8540ff8d39cae4991da801, State: Initialized, Role: FOLLOWER
I20260925 15:16:27.111194 18159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0f43360d8540ff8d39cae4991da801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38411 } }
I20260925 15:16:27.111361 18159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:27.111384 18159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:27.111419 18159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:27.112149 18159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0f43360d8540ff8d39cae4991da801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38411 } }
I20260925 15:16:27.112229 18159 leader_election.cc:302] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a0f43360d8540ff8d39cae4991da801; no voters:
I20260925 15:16:27.112336 18159 leader_election.cc:288] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:16:27.112401 18164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:27.112545 18164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [term 1 LEADER]: Becoming Leader. State: Replica: 3a0f43360d8540ff8d39cae4991da801, State: Running, Role: LEADER
I20260925 15:16:27.112587 18159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:27.112641 18164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0f43360d8540ff8d39cae4991da801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38411 } }
I20260925 15:16:27.113001 18165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a0f43360d8540ff8d39cae4991da801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0f43360d8540ff8d39cae4991da801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38411 } } }
I20260925 15:16:27.113009 18166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a0f43360d8540ff8d39cae4991da801. Latest consensus state: current_term: 1 leader_uuid: "3a0f43360d8540ff8d39cae4991da801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a0f43360d8540ff8d39cae4991da801" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38411 } } }
I20260925 15:16:27.113134 18165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:27.113140 18166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:27.113687 18173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:27.114128 18173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:27.115720 18173 catalog_manager.cc:1347] Generated new cluster ID: 4452be6778ba4fb0b041ac56a34c9e1e
I20260925 15:16:27.115774 18173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:27.126709 18173 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:27.126868 18173 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:27.134790 18173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a0f43360d8540ff8d39cae4991da801: Generated new TSK 0
I20260925 15:16:27.134968 18173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:16:27.141335 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38411
--builtin_ntp_servers=127.8.43.148:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:16:27.222157 18183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:27.222393 18183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:27.222431 18183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:27.223680 18183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:27.223740 18183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:27.223783 18183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:16:27.225154 18183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:27.225419 18183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:27.225710 18183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:27.231352 18189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:27.231403 18183 server_base.cc:1034] running on GCE node
W20260925 15:16:27.231354 18188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:27.231578 18191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:27.231778 18183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:27.232045 18183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:27.233196 18183 hybrid_clock.cc:648] HybridClock initialized: now 1790349387233172 us; error 38 us; skew 500 ppm
I20260925 15:16:27.234467 18183 webserver.cc:460] Webserver started at http://127.8.43.129:43867/ using document root <none> and password file <none>
I20260925 15:16:27.234704 18183 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:27.234772 18183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:27.234920 18183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:27.236047 18183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "4c0e6b57a0ed4d2ab38ee08fb67c3a97"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "d77d456744b5701f4ac32dd31be7ab64"
server_key_iv: "4d2bbf488d1e8e82cdfd4f880ab62af8"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.236409 18183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "4c0e6b57a0ed4d2ab38ee08fb67c3a97"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "d77d456744b5701f4ac32dd31be7ab64"
server_key_iv: "4d2bbf488d1e8e82cdfd4f880ab62af8"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.237715 18183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:16:27.238458 18197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.238672 18183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:27.238763 18183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "4c0e6b57a0ed4d2ab38ee08fb67c3a97"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "d77d456744b5701f4ac32dd31be7ab64"
server_key_iv: "4d2bbf488d1e8e82cdfd4f880ab62af8"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.238842 18183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:27.243355 18183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:27.243640 18183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:27.243757 18183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:27.243917 18183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:27.244241 18183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:27.244284 18183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.244328 18183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:27.244347 18183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.250315 18183 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:37237
I20260925 15:16:27.250372 18310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:37237 every 8 connection(s)
I20260925 15:16:27.250677 18183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:27.253578 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.251486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:35087 (local address 127.8.43.190:38411)
0925 15:16:27.251549 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:27.251552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:27.251788 (+ 236us) server_negotiation.cc:407] Connection header received
0925 15:16:27.252223 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:27.252224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:27.252229 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:27.252289 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:27.253372 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:27.253373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:27.253373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:16:27.253377 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:27.253391 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:16:27.253395 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:27.253441 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:27.253514 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:16:27.253526 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:16:27.253741 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.251484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38411 (local address 127.8.43.129:35087)
0925 15:16:27.251651 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:27.251685 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:27.252137 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:27.252358 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:27.252362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:27.252370 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:27.253233 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:16:27.253233 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:27.253261 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:16:27.253290 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:27.253439 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:27.253441 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:27.253443 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:16:27.253507 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:16:27.253553 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260925 15:16:27.254281 18311 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38411
I20260925 15:16:27.254382 18311 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:27.254539 18311 heartbeater.cc:511] Master 127.8.43.190:38411 requested a full tablet report, sending...
I20260925 15:16:27.255021 18123 ts_manager.cc:194] Registered new tserver with Master: 4c0e6b57a0ed4d2ab38ee08fb67c3a97 (127.8.43.129:37237)
I20260925 15:16:27.256330 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 18183
I20260925 15:16:27.256409 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:16:27.256563 8366 external_mini_cluster.cc:1428] Setting key fd576f4d6e9f5a3560e907f931cd814e
I20260925 15:16:27.257449 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38411
--builtin_ntp_servers=127.8.43.148:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:16:27.337316 18314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:27.337533 18314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:27.337556 18314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:27.338642 18314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:27.338675 18314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:27.338698 18314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:16:27.339950 18314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:27.340126 18314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:27.340340 18314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:27.345222 18319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:27.345239 18320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:27.345227 18322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:27.345496 18314 server_base.cc:1034] running on GCE node
I20260925 15:16:27.345649 18314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:27.345829 18314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:27.346961 18314 hybrid_clock.cc:648] HybridClock initialized: now 1790349387346936 us; error 44 us; skew 500 ppm
I20260925 15:16:27.348136 18314 webserver.cc:460] Webserver started at http://127.8.43.130:40839/ using document root <none> and password file <none>
I20260925 15:16:27.348366 18314 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:27.348410 18314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:27.348559 18314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:27.349674 18314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "4c2dca4b72d240f0bc1a96f19cfe502d"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "77c912306c2f655ae3ca6e4e5af697e9"
server_key_iv: "c5aa5e8aeb3e19562a5f5af1cc0eb77c"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.350015 18314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "4c2dca4b72d240f0bc1a96f19cfe502d"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "77c912306c2f655ae3ca6e4e5af697e9"
server_key_iv: "c5aa5e8aeb3e19562a5f5af1cc0eb77c"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.351339 18314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:16:27.352035 18328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.352221 18314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:27.352309 18314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "4c2dca4b72d240f0bc1a96f19cfe502d"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "77c912306c2f655ae3ca6e4e5af697e9"
server_key_iv: "c5aa5e8aeb3e19562a5f5af1cc0eb77c"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.352378 18314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:27.355665 18314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:27.355921 18314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:27.356048 18314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:27.356259 18314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:27.356534 18314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:27.356564 18314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.356612 18314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:27.356640 18314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.362290 18314 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:33157
I20260925 15:16:27.362346 18441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:33157 every 8 connection(s)
I20260925 15:16:27.362625 18314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:27.365339 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.363335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:53563 (local address 127.8.43.190:38411)
0925 15:16:27.363376 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:27.363378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:27.363613 (+ 235us) server_negotiation.cc:407] Connection header received
0925 15:16:27.364044 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:27.364045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:27.364050 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:27.364102 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:27.365126 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:27.365127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:27.365128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:16:27.365132 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:27.365157 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:16:27.365161 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:27.365222 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:27.365269 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:16:27.365281 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:16:27.365540 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.363349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38411 (local address 127.8.43.130:53563)
0925 15:16:27.363502 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:27.363532 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:27.363964 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:27.364150 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:27.364153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:27.364160 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:27.364972 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:16:27.364973 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:27.364998 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:16:27.365024 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:27.365222 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:27.365224 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:27.365226 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:16:27.365275 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:16:27.365319 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260925 15:16:27.366024 18442 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38411
I20260925 15:16:27.366093 18442 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:27.366238 18442 heartbeater.cc:511] Master 127.8.43.190:38411 requested a full tablet report, sending...
I20260925 15:16:27.366598 18123 ts_manager.cc:194] Registered new tserver with Master: 4c2dca4b72d240f0bc1a96f19cfe502d (127.8.43.130:33157)
I20260925 15:16:27.372114 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 18314
I20260925 15:16:27.372196 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:16:27.372390 8366 external_mini_cluster.cc:1428] Setting key 5de3381a46054f70c9e0446470dcbdc3
I20260925 15:16:27.373436 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38411
--builtin_ntp_servers=127.8.43.148:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:16:27.454394 18445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:27.454634 18445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:27.454680 18445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:27.455919 18445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:27.455981 18445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:27.456024 18445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:16:27.457363 18445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:27.457641 18445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:27.457935 18445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:27.462657 18450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:27.462721 18453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:27.462713 18451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:27.462967 18445 server_base.cc:1034] running on GCE node
I20260925 15:16:27.463165 18445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:27.463424 18445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:27.464589 18445 hybrid_clock.cc:648] HybridClock initialized: now 1790349387464579 us; error 29 us; skew 500 ppm
I20260925 15:16:27.465734 18445 webserver.cc:460] Webserver started at http://127.8.43.131:46375/ using document root <none> and password file <none>
I20260925 15:16:27.465969 18445 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:27.466034 18445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:27.466166 18445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:27.467164 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "a5620c06b173421989695fef56e53e96"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "2325d19605994010a10894a451e909ca"
server_key_iv: "57dbee15a953722ec7ba372f8d997cc7"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.467573 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "a5620c06b173421989695fef56e53e96"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "2325d19605994010a10894a451e909ca"
server_key_iv: "57dbee15a953722ec7ba372f8d997cc7"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.468842 18445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:27.469551 18459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.469712 18445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:27.469811 18445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "a5620c06b173421989695fef56e53e96"
format_stamp: "Formatted at 2026-09-25 15:16:27 on dist-test-slave-c5d6"
server_key: "2325d19605994010a10894a451e909ca"
server_key_iv: "57dbee15a953722ec7ba372f8d997cc7"
server_key_version: "encryptionkey@0"
I20260925 15:16:27.469892 18445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:27.473097 18445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:27.473354 18445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:27.473488 18445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:27.473683 18445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:27.473964 18445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:27.474014 18445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.474050 18445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:27.474078 18445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:27.479403 18445 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:38985
I20260925 15:16:27.479432 18572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:38985 every 8 connection(s)
I20260925 15:16:27.479736 18445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:27.482292 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.480448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:57179 (local address 127.8.43.190:38411)
0925 15:16:27.480508 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:27.480510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:27.480700 (+ 190us) server_negotiation.cc:407] Connection header received
0925 15:16:27.481086 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:27.481087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:27.481091 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:27.481143 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:27.482123 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:27.482124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:27.482124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:16:27.482128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:27.482141 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:16:27.482145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:27.482188 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:27.482231 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:16:27.482242 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:16:27.482515 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.480415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38411 (local address 127.8.43.131:57179)
0925 15:16:27.480572 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:27.480610 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:27.481011 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:27.481188 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:27.481190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:27.481198 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:27.481994 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:16:27.481995 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:27.482020 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:16:27.482045 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:27.482186 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:27.482188 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:27.482190 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:16:27.482238 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 15:16:27.482280 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260925 15:16:27.482982 18573 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38411
I20260925 15:16:27.483078 18573 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:27.483256 18573 heartbeater.cc:511] Master 127.8.43.190:38411 requested a full tablet report, sending...
I20260925 15:16:27.483573 18123 ts_manager.cc:194] Registered new tserver with Master: a5620c06b173421989695fef56e53e96 (127.8.43.131:38985)
I20260925 15:16:27.488082 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 18445
I20260925 15:16:27.488160 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:16:27.488368 8366 external_mini_cluster.cc:1428] Setting key 090ffbbc2fb36a3a8b22be8e7bc323e0
I20260925 15:16:27.489312 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:16:27.490986 18162 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:16:27.490554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.8.43.190:38411)
0925 15:16:27.490613 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:27.490615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:27.490726 (+ 111us) server_negotiation.cc:407] Connection header received
0925 15:16:27.490771 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:27.490771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:27.490775 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:27.490842 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:27.490925 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38802: BlockingRecv error: recv got EOF from 127.0.0.1:38802 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:16:27.491108 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 18183
I20260925 15:16:27.495719 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 18314
I20260925 15:16:27.500310 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 18445
I20260925 15:16:27.505926 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 18092
2026-09-25T15:16:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:16:27 dist-test-slave-c5d6 krb5kdc[18583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:16:27 dist-test-slave-c5d6 krb5kdc[18583](info): set up 2 sockets
Sep 25 15:16:27 dist-test-slave-c5d6 krb5kdc[18583](info): commencing operation
krb5kdc: starting...
W20260925 15:16:28.559715 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:16:28 dist-test-slave-c5d6 krb5kdc[18583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:16:28.687050 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:37019
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:28.769097 18596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:28.769300 18596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:28.769322 18596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:28.770465 18596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:28.770504 18596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:28.770519 18596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:28.770531 18596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:28.770543 18596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:28.772089 18596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:28.772293 18596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:28.772469 18596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:28.777431 18602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:28.777470 18601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:28.777709 18596 server_base.cc:1034] running on GCE node
W20260925 15:16:28.777489 18604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:28.778040 18596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:28.778332 18596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:28.779500 18596 hybrid_clock.cc:648] HybridClock initialized: now 1790349388779491 us; error 34 us; skew 500 ppm
Sep 25 15:16:28 dist-test-slave-c5d6 krb5kdc[18583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349388, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:28.781939 18596 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:16:28.782315 18596 webserver.cc:460] Webserver started at http://127.8.43.190:45649/ using document root <none> and password file <none>
I20260925 15:16:28.782564 18596 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:28.782609 18596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:28.782752 18596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:28.783916 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "e3eab5ebcb0546b69be9273e72d1306c"
format_stamp: "Formatted at 2026-09-25 15:16:28 on dist-test-slave-c5d6"
server_key: "5c63ea491bdc4de2b5e41343c15026eb"
server_key_iv: "2af8b972ba37f2943af75aa92cceb328"
server_key_version: "encryptionkey@0"
I20260925 15:16:28.784265 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "e3eab5ebcb0546b69be9273e72d1306c"
format_stamp: "Formatted at 2026-09-25 15:16:28 on dist-test-slave-c5d6"
server_key: "5c63ea491bdc4de2b5e41343c15026eb"
server_key_iv: "2af8b972ba37f2943af75aa92cceb328"
server_key_version: "encryptionkey@0"
I20260925 15:16:28.785575 18596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:16:28.786343 18611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:28.786535 18596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:16:28.786638 18596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "e3eab5ebcb0546b69be9273e72d1306c"
format_stamp: "Formatted at 2026-09-25 15:16:28 on dist-test-slave-c5d6"
server_key: "5c63ea491bdc4de2b5e41343c15026eb"
server_key_iv: "2af8b972ba37f2943af75aa92cceb328"
server_key_version: "encryptionkey@0"
I20260925 15:16:28.786715 18596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:28.802067 18596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:28.804026 18596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:28.804188 18596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:28.808035 18596 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:37019
I20260925 15:16:28.808073 18663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:37019 every 8 connection(s)
I20260925 15:16:28.808445 18596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:28.809386 18664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:28.811625 18664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c: Bootstrap starting.
I20260925 15:16:28.812191 18664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:28.812201 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 18596
I20260925 15:16:28.812304 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:16:28.812481 8366 external_mini_cluster.cc:1428] Setting key 7649c06331f667c89fce3969eb7a0cc1
I20260925 15:16:28.812495 18664 log.cc:824] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:28.813242 18664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c: No bootstrap required, opened a new log
I20260925 15:16:28.814776 18664 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3eab5ebcb0546b69be9273e72d1306c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37019 } }
I20260925 15:16:28.814915 18664 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:28.814939 18664 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3eab5ebcb0546b69be9273e72d1306c, State: Initialized, Role: FOLLOWER
I20260925 15:16:28.815073 18664 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3eab5ebcb0546b69be9273e72d1306c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37019 } }
I20260925 15:16:28.815171 18664 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:28.815212 18664 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:28.815274 18664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:28.816279 18664 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3eab5ebcb0546b69be9273e72d1306c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37019 } }
I20260925 15:16:28.816386 18664 leader_election.cc:302] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3eab5ebcb0546b69be9273e72d1306c; no voters:
I20260925 15:16:28.816543 18664 leader_election.cc:288] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:16:28.816612 18669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:28.816751 18669 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [term 1 LEADER]: Becoming Leader. State: Replica: e3eab5ebcb0546b69be9273e72d1306c, State: Running, Role: LEADER
I20260925 15:16:28.816769 18664 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:28.817682 18669 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3eab5ebcb0546b69be9273e72d1306c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37019 } }
I20260925 15:16:28.818280 18671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3eab5ebcb0546b69be9273e72d1306c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3eab5ebcb0546b69be9273e72d1306c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37019 } } }
I20260925 15:16:28.818375 18671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:28.818423 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3eab5ebcb0546b69be9273e72d1306c. Latest consensus state: current_term: 1 leader_uuid: "e3eab5ebcb0546b69be9273e72d1306c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3eab5ebcb0546b69be9273e72d1306c" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 37019 } } }
I20260925 15:16:28.818495 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c [sys.catalog]: This master's current role is: LEADER
W20260925 15:16:28.819082 18683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:16:28.819171 18683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:16:28.819288 18684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:28.819476 18684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:28.820926 18684 catalog_manager.cc:1347] Generated new cluster ID: 68ec404b0a854619acacb61bc69f0cc3
I20260925 15:16:28.820977 18684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:16:28 dist-test-slave-c5d6 krb5kdc[18583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:16:28.825182 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:28.813015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.8.43.190:37019)
0925 15:16:28.813189 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:28.813196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:28.815206 (+ 2010us) server_negotiation.cc:407] Connection header received
0925 15:16:28.815404 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:28.815413 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:28.815470 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:28.815619 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:28.816197 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:28.817544 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:28.818675 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:28.819080 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:28.819206 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:28.821840 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:28.821844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:28.821845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:28.821856 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:28.823621 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:28.823979 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:28.823982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:28.823982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:28.824128 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:28.824347 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:28.824348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:28.824348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:28.824590 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:28.824755 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:28.824856 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:16:28.824994 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260925 15:16:28.831578 18684 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:28.832257 18684 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:28.841003 18684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c: Generated new TSK 0
I20260925 15:16:28.841176 18684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:16:28.872337 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:37019
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:28.964509 18690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:28.964720 18690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:28.964743 18690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:28.965901 18690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:28.965945 18690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:28.965972 18690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:16:28.967571 18690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:28.967818 18690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:28.968070 18690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:28.973065 18696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:28.973141 18698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:28.973078 18695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:28.973325 18690 server_base.cc:1034] running on GCE node
I20260925 15:16:28.973604 18690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:28.973879 18690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:28.975036 18690 hybrid_clock.cc:648] HybridClock initialized: now 1790349388975026 us; error 27 us; skew 500 ppm
Sep 25 15:16:28 dist-test-slave-c5d6 krb5kdc[18583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349388, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:28.977622 18690 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:16:28.978096 18690 webserver.cc:460] Webserver started at http://127.8.43.129:35945/ using document root <none> and password file <none>
I20260925 15:16:28.978281 18690 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:28.978338 18690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:28.978433 18690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:28.979523 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "e8915a8167a342c0b2e6355b33731b2d"
format_stamp: "Formatted at 2026-09-25 15:16:28 on dist-test-slave-c5d6"
server_key: "6894c0cfb5f75145da99d176066dc63b"
server_key_iv: "ddec925caebe2a2cda33e35f7c8251b0"
server_key_version: "encryptionkey@0"
I20260925 15:16:28.979826 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "e8915a8167a342c0b2e6355b33731b2d"
format_stamp: "Formatted at 2026-09-25 15:16:28 on dist-test-slave-c5d6"
server_key: "6894c0cfb5f75145da99d176066dc63b"
server_key_iv: "ddec925caebe2a2cda33e35f7c8251b0"
server_key_version: "encryptionkey@0"
I20260925 15:16:28.981163 18690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:16:28.981885 18705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:28.982070 18690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:16:28.982156 18690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "e8915a8167a342c0b2e6355b33731b2d"
format_stamp: "Formatted at 2026-09-25 15:16:28 on dist-test-slave-c5d6"
server_key: "6894c0cfb5f75145da99d176066dc63b"
server_key_iv: "ddec925caebe2a2cda33e35f7c8251b0"
server_key_version: "encryptionkey@0"
I20260925 15:16:28.982216 18690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:29.004056 18690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:29.006111 18690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:29.006233 18690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:29.006421 18690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:29.006749 18690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:29.006784 18690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:29.006837 18690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:29.006865 18690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:29.012626 18690 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:43399
I20260925 15:16:29.012729 18818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:43399 every 8 connection(s)
I20260925 15:16:29.013037 18690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:29.017493 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 18690
I20260925 15:16:29.017588 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:16:29.017831 8366 external_mini_cluster.cc:1428] Setting key 42beeae59fdd7b6ff0b3fb5c2c47ec11
Sep 25 15:16:29 dist-test-slave-c5d6 krb5kdc[18583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349388, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:16:29.023317 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.013782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:42139 (local address 127.8.43.190:37019)
0925 15:16:29.013828 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.013830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.014023 (+ 193us) server_negotiation.cc:407] Connection header received
0925 15:16:29.014556 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.014558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.014572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.014638 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:29.015642 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.016151 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.017251 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.017540 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.017636 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.020186 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:29.020187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:29.020187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:29.020194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:29.021702 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:29.022228 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:29.022229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:29.022229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:29.022281 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:29.022575 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:29.022575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:29.022576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:29.022688 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:29.022738 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.023094 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.023170 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:16:29.023593 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.013739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:37019 (local address 127.8.43.129:42139)
0925 15:16:29.013890 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.013933 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:29.014428 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:29.014693 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:29.014697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:29.015339 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:29.015536 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.015541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.016236 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.016237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.017159 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.017162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.017653 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.017654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.017818 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.018418 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:29.018427 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:29.020092 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:29.021797 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:29.021804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:29.021807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:29.022146 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:29.022363 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:29.022364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:29.022365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:29.022509 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:29.022753 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:29.022760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:29.022936 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 15:16:29.023063 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:16:29.023278 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260925 15:16:29.024166 18819 heartbeater.cc:347] Connected to a master server at 127.8.43.190:37019
I20260925 15:16:29.024269 18819 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:29.024494 18819 heartbeater.cc:511] Master 127.8.43.190:37019 requested a full tablet report, sending...
I20260925 15:16:29.025076 18628 ts_manager.cc:194] Registered new tserver with Master: e8915a8167a342c0b2e6355b33731b2d (127.8.43.129:43399)
I20260925 15:16:29.025693 18628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:42139
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:16:29.043529 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:37019
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:29.125116 18824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:29.125316 18824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:29.125340 18824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:29.126506 18824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:29.126552 18824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:29.126580 18824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:16:29.128026 18824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:29.128268 18824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:29.128510 18824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:29.133375 18832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:29.133486 18829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:29.133375 18830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:29.133505 18824 server_base.cc:1034] running on GCE node
I20260925 15:16:29.133893 18824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:29.134122 18824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:29.135273 18824 hybrid_clock.cc:648] HybridClock initialized: now 1790349389135266 us; error 38 us; skew 500 ppm
Sep 25 15:16:29 dist-test-slave-c5d6 krb5kdc[18583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349389, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:29.137744 18824 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:16:29.138152 18824 webserver.cc:460] Webserver started at http://127.8.43.130:45721/ using document root <none> and password file <none>
I20260925 15:16:29.138361 18824 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:29.138424 18824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:29.138562 18824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:29.139613 18824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "8c89f88cb91240e2878af5554580526c"
format_stamp: "Formatted at 2026-09-25 15:16:29 on dist-test-slave-c5d6"
server_key: "03424678ca707509a2374b9f5334d8b2"
server_key_iv: "c5a0843c35637d3fd5934360881f41ff"
server_key_version: "encryptionkey@0"
I20260925 15:16:29.140004 18824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "8c89f88cb91240e2878af5554580526c"
format_stamp: "Formatted at 2026-09-25 15:16:29 on dist-test-slave-c5d6"
server_key: "03424678ca707509a2374b9f5334d8b2"
server_key_iv: "c5a0843c35637d3fd5934360881f41ff"
server_key_version: "encryptionkey@0"
I20260925 15:16:29.141248 18824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:16:29.141952 18839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:29.142115 18824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:29.142208 18824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "8c89f88cb91240e2878af5554580526c"
format_stamp: "Formatted at 2026-09-25 15:16:29 on dist-test-slave-c5d6"
server_key: "03424678ca707509a2374b9f5334d8b2"
server_key_iv: "c5a0843c35637d3fd5934360881f41ff"
server_key_version: "encryptionkey@0"
I20260925 15:16:29.142274 18824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:29.151458 18824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:29.153110 18824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:29.153252 18824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:29.153450 18824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:29.153723 18824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:29.153771 18824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:29.153808 18824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:29.153843 18824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:29.159497 18824 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:40111
I20260925 15:16:29.159544 18952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:40111 every 8 connection(s)
I20260925 15:16:29.159869 18824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:16:29 dist-test-slave-c5d6 krb5kdc[18583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349389, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:16:29.167994 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 18824
I20260925 15:16:29.168081 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:16:29.168277 8366 external_mini_cluster.cc:1428] Setting key 29686c52e05a5f23881d61b5791ef298
I20260925 15:16:29.168391 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.160516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:37475 (local address 127.8.43.190:37019)
0925 15:16:29.160591 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.160593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.160742 (+ 149us) server_negotiation.cc:407] Connection header received
0925 15:16:29.161185 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.161186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.161196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.161247 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:29.162103 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.162595 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.163591 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.163786 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.163862 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.165873 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:29.165874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:29.165874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:29.165880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:29.167056 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:29.167476 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:29.167477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:29.167477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:29.167525 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:29.167723 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:29.167723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:29.167723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:29.167859 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:29.167941 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.168195 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.168283 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:16:29.168525 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.160481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:37019 (local address 127.8.43.130:37475)
0925 15:16:29.160628 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.160662 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:29.161103 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:29.161294 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:29.161296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:29.161857 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:29.162030 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.162034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.162670 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.162670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.163523 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.163525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.163873 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.163874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.163984 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.164494 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:29.164502 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:29.165790 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:29.167132 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:29.167138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:29.167140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:29.167415 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:29.167583 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:29.167584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:29.167584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:29.167675 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:29.167923 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:29.167926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:29.168028 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 15:16:29.168156 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 15:16:29.168312 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":91,"threads_started":1}
I20260925 15:16:29.169078 18953 heartbeater.cc:347] Connected to a master server at 127.8.43.190:37019
I20260925 15:16:29.169164 18953 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:29.169348 18953 heartbeater.cc:511] Master 127.8.43.190:37019 requested a full tablet report, sending...
I20260925 15:16:29.169729 18628 ts_manager.cc:194] Registered new tserver with Master: 8c89f88cb91240e2878af5554580526c (127.8.43.130:40111)
I20260925 15:16:29.170113 18628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:37475
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:16:29.192058 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:37019
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:32781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:29.272450 18958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:29.272665 18958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:29.272687 18958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:29.273815 18958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:29.273854 18958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:29.273880 18958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:16:29.275403 18958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:32781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:37019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:16:29.275612 18958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:29.275796 18958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:29.280966 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:29.280932 18964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:29.281008 18963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:29.281031 18958 server_base.cc:1034] running on GCE node
I20260925 15:16:29.281358 18958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:29.281625 18958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:29.282786 18958 hybrid_clock.cc:648] HybridClock initialized: now 1790349389282770 us; error 28 us; skew 500 ppm
Sep 25 15:16:29 dist-test-slave-c5d6 krb5kdc[18583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349389, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:29.285543 18958 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:16:29.286031 18958 webserver.cc:460] Webserver started at http://127.8.43.131:36811/ using document root <none> and password file <none>
I20260925 15:16:29.286283 18958 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:29.286356 18958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:29.286527 18958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:29.287693 18958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "7ece50ef90be40d6a5e9dc5393d70458"
format_stamp: "Formatted at 2026-09-25 15:16:29 on dist-test-slave-c5d6"
server_key: "33e3fe5a1a83b492279999c923214a80"
server_key_iv: "c51f3dccb7b8e9b3a829654c69d4ee31"
server_key_version: "encryptionkey@0"
I20260925 15:16:29.288090 18958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "7ece50ef90be40d6a5e9dc5393d70458"
format_stamp: "Formatted at 2026-09-25 15:16:29 on dist-test-slave-c5d6"
server_key: "33e3fe5a1a83b492279999c923214a80"
server_key_iv: "c51f3dccb7b8e9b3a829654c69d4ee31"
server_key_version: "encryptionkey@0"
I20260925 15:16:29.289527 18958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:16:29.290328 18973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:29.290504 18958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:16:29.290604 18958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "7ece50ef90be40d6a5e9dc5393d70458"
format_stamp: "Formatted at 2026-09-25 15:16:29 on dist-test-slave-c5d6"
server_key: "33e3fe5a1a83b492279999c923214a80"
server_key_iv: "c51f3dccb7b8e9b3a829654c69d4ee31"
server_key_version: "encryptionkey@0"
I20260925 15:16:29.290694 18958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:29.308074 18958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:29.310583 18958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:29.310760 18958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:29.310989 18958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:29.311344 18958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:29.311398 18958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:29.311439 18958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:29.311470 18958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:29.318291 18958 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:44219
I20260925 15:16:29.318356 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:44219 every 8 connection(s)
I20260925 15:16:29.318662 18958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:16:29 dist-test-slave-c5d6 krb5kdc[18583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349389, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:16:29.326582 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 18958
I20260925 15:16:29.326689 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:16:29.326910 8366 external_mini_cluster.cc:1428] Setting key 19c9d47030a99eb80db3b3e3090b60aa
I20260925 15:16:29.328502 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.319448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:34493 (local address 127.8.43.190:37019)
0925 15:16:29.319496 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.319498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.319730 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:16:29.320281 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.320282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.320293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.320349 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:29.321311 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.321792 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.322883 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.323065 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.323120 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.325300 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:29.325302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:29.325302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:29.325312 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:29.326908 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:29.327483 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:29.327484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:29.327485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:29.327553 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:29.327843 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:29.327844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:29.327845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:29.327972 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:29.328055 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.328328 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.328407 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:16:29.328689 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.319442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:37019 (local address 127.8.43.131:34493)
0925 15:16:29.319603 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.319638 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:29.320189 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:29.320402 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:29.320406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:29.321010 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:29.321203 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.321208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.321909 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.321910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.322767 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.322769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.323138 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.323139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.323293 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.323751 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:29.323759 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:29.325201 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:29.327008 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:29.327016 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:29.327017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:29.327408 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:29.327625 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:29.327626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:29.327627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:29.327747 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:29.328044 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:29.328051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:29.328190 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 15:16:29.328306 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:16:29.328462 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260925 15:16:29.329219 19087 heartbeater.cc:347] Connected to a master server at 127.8.43.190:37019
I20260925 15:16:29.329299 19087 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:29.329487 19087 heartbeater.cc:511] Master 127.8.43.190:37019 requested a full tablet report, sending...
I20260925 15:16:29.329861 18628 ts_manager.cc:194] Registered new tserver with Master: 7ece50ef90be40d6a5e9dc5393d70458 (127.8.43.131:44219)
I20260925 15:16:29.330278 18628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:34493
I20260925 15:16:29.338668 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:16:29.345615 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.339942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.8.43.190:37019)
0925 15:16:29.339997 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.340000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.340148 (+ 148us) server_negotiation.cc:407] Connection header received
0925 15:16:29.340189 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.340190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.340198 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.340248 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:29.340720 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.341337 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.342067 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.342245 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.342312 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.343402 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:29.343403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:29.343404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:29.343410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:29.344432 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:29.344838 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:29.344839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:29.344840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:29.344897 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:29.345091 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:29.345092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:29.345093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:29.345232 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:29.345338 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.345449 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.345520 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:16:29.347342 18628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:29.348884 18628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:29.355324 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.351157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:44219 (local address 127.0.0.1:44040)
0925 15:16:29.351437 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.351446 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:29.351508 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:29.351759 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:29.351762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:29.351764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:29.351823 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.351825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.353102 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.353102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.354371 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.354373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.355022 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.355023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.355098 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.355110 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:16:29.355180 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:16:29.355260 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":43,"thread_start_us":112,"threads_started":1}
I20260925 15:16:29.355337 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.351278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.8.43.131:44219)
0925 15:16:29.351429 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.351433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.351493 (+ 60us) server_negotiation.cc:407] Connection header received
0925 15:16:29.351628 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.351632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.351650 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.351736 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:29.351912 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.352993 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.354458 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.354934 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.355023 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.355080 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.355194 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.355253 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260925 15:16:29.355656 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.351415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:40111 (local address 127.0.0.1:52338)
0925 15:16:29.351535 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.351544 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:29.351583 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:29.352328 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:29.352329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:29.352331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:29.352363 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.352364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.353469 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.353470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.354381 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.354383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.355395 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.355397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.355476 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.355481 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:16:29.355545 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:16:29.355582 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260925 15:16:29.355685 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.351892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.8.43.130:40111)
0925 15:16:29.352174 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.352178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.352190 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:16:29.352229 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.352231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.352247 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.352332 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:29.352434 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.353369 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.354499 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.355196 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.355299 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.355356 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.355562 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.355612 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":94,"threads_started":1}
I20260925 15:16:29.356025 19021 tablet_service.cc:1467] Processing CreateTablet for tablet 69aff59d1728425aa1a948429635ccd1 (DEFAULT_TABLE table=test-table [id=513590aa2c4341a9ab92fc2239065713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:29.356376 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69aff59d1728425aa1a948429635ccd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:29.356379 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.351578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.8.43.129:43399)
0925 15:16:29.351818 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.351823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.351832 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:16:29.351868 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.351870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.351886 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.351962 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:29.352055 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.352917 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.354457 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.355191 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.355295 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.355356 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.356209 (+ 853us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.356276 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260925 15:16:29.356477 18887 tablet_service.cc:1467] Processing CreateTablet for tablet 69aff59d1728425aa1a948429635ccd1 (DEFAULT_TABLE table=test-table [id=513590aa2c4341a9ab92fc2239065713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:29.356771 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69aff59d1728425aa1a948429635ccd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:29.356822 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.351121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:43399 (local address 127.0.0.1:37788)
0925 15:16:29.351348 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.351399 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:29.351464 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:29.351966 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:29.351967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:29.351968 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:29.352005 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.352006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.353013 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.353014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.354343 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.354346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.355685 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.355686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.356100 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.356105 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:16:29.356175 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:16:29.356746 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":33,"thread_start_us":83,"threads_started":1}
I20260925 15:16:29.357515 18753 tablet_service.cc:1467] Processing CreateTablet for tablet 69aff59d1728425aa1a948429635ccd1 (DEFAULT_TABLE table=test-table [id=513590aa2c4341a9ab92fc2239065713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:29.357836 18753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69aff59d1728425aa1a948429635ccd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:29.359714 19104 tablet_bootstrap.cc:492] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Bootstrap starting.
I20260925 15:16:29.360127 19105 tablet_bootstrap.cc:492] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Bootstrap starting.
I20260925 15:16:29.360441 19104 tablet_bootstrap.cc:654] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:29.360742 19106 tablet_bootstrap.cc:492] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d: Bootstrap starting.
I20260925 15:16:29.360782 19105 tablet_bootstrap.cc:654] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:29.361114 19104 log.cc:824] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:29.361367 19105 log.cc:824] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:29.361478 19106 tablet_bootstrap.cc:654] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:29.361879 19104 tablet_bootstrap.cc:492] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: No bootstrap required, opened a new log
I20260925 15:16:29.361938 19104 ts_tablet_manager.cc:1397] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:16:29.361982 19106 log.cc:824] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:29.362646 19106 tablet_bootstrap.cc:492] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d: No bootstrap required, opened a new log
I20260925 15:16:29.362712 19106 ts_tablet_manager.cc:1397] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:16:29.363646 19104 raft_consensus.cc:348] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.363852 19104 raft_consensus.cc:374] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:29.363871 19105 tablet_bootstrap.cc:492] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: No bootstrap required, opened a new log
I20260925 15:16:29.363894 19104 raft_consensus.cc:729] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ece50ef90be40d6a5e9dc5393d70458, State: Initialized, Role: FOLLOWER
I20260925 15:16:29.363931 19105 ts_tablet_manager.cc:1397] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 15:16:29.364018 19104 consensus_queue.cc:260] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.364300 19104 ts_tablet_manager.cc:1428] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:16:29.364455 19087 heartbeater.cc:503] Master 127.8.43.190:37019 was elected leader, sending a full tablet report...
I20260925 15:16:29.364463 19106 raft_consensus.cc:348] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.364653 19106 raft_consensus.cc:374] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:29.364696 19106 raft_consensus.cc:729] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8915a8167a342c0b2e6355b33731b2d, State: Initialized, Role: FOLLOWER
I20260925 15:16:29.364826 19106 consensus_queue.cc:260] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.365082 19106 ts_tablet_manager.cc:1428] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:16:29.365140 18819 heartbeater.cc:503] Master 127.8.43.190:37019 was elected leader, sending a full tablet report...
I20260925 15:16:29.366113 19105 raft_consensus.cc:348] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.366233 19105 raft_consensus.cc:374] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:29.366283 19105 raft_consensus.cc:729] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c89f88cb91240e2878af5554580526c, State: Initialized, Role: FOLLOWER
I20260925 15:16:29.366397 19105 consensus_queue.cc:260] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.366668 19105 ts_tablet_manager.cc:1428] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:29.366686 18953 heartbeater.cc:503] Master 127.8.43.190:37019 was elected leader, sending a full tablet report...
W20260925 15:16:29.410425 18954 tablet.cc:2369] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:29.513779 18820 tablet.cc:2369] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:29.569307 19088 tablet.cc:2369] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:29.618500 19112 raft_consensus.cc:482] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:29.618631 19112 raft_consensus.cc:504] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.619115 19112 leader_election.cc:288] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8915a8167a342c0b2e6355b33731b2d (127.8.43.129:43399), 7ece50ef90be40d6a5e9dc5393d70458 (127.8.43.131:44219)
I20260925 15:16:29.622161 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.619414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:48513 (local address 127.8.43.129:43399)
0925 15:16:29.619460 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.619461 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.619472 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:16:29.619502 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.619503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.619515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.619575 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:29.619783 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.620327 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.621420 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.621788 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.621844 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.621851 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.622044 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.622081 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:16:29.622144 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.619316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:43399 (local address 127.8.43.130:48513)
0925 15:16:29.619363 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.619376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:29.619428 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:29.619614 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:29.619615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:29.619617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:29.619703 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.619706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.620416 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.620417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.621329 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.621331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.621877 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.621878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.621965 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.621969 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:16:29.622028 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:16:29.622080 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 15:16:29.622325 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.619461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:42553 (local address 127.8.43.131:44219)
0925 15:16:29.619510 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.619512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.619621 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:16:29.619653 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.619654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.619664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.619739 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:29.619950 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.620497 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.621469 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.621941 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.622022 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.622032 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.622204 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.622247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:16:29.622488 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.619317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:44219 (local address 127.8.43.130:42553)
0925 15:16:29.619544 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.619553 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:29.619601 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:29.619764 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:29.619765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:29.619767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:29.619866 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.619868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.620579 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.620579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.621352 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:29.621354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.622048 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.622049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:29.622115 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.622119 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:16:29.622176 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:16:29.622355 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":40,"thread_start_us":116,"threads_started":1}
I20260925 15:16:29.622560 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aff59d1728425aa1a948429635ccd1" candidate_uuid: "8c89f88cb91240e2878af5554580526c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8915a8167a342c0b2e6355b33731b2d" is_pre_election: true
I20260925 15:16:29.622735 18773 raft_consensus.cc:2454] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c89f88cb91240e2878af5554580526c in term 0.
I20260925 15:16:29.622830 19041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aff59d1728425aa1a948429635ccd1" candidate_uuid: "8c89f88cb91240e2878af5554580526c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ece50ef90be40d6a5e9dc5393d70458" is_pre_election: true
I20260925 15:16:29.622969 19041 raft_consensus.cc:2454] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c89f88cb91240e2878af5554580526c in term 0.
I20260925 15:16:29.623034 18843 leader_election.cc:302] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c89f88cb91240e2878af5554580526c, e8915a8167a342c0b2e6355b33731b2d; no voters:
I20260925 15:16:29.623209 19112 raft_consensus.cc:2781] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:29.623353 19112 raft_consensus.cc:482] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:29.623391 19112 raft_consensus.cc:3037] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:29.624114 19112 raft_consensus.cc:504] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.624353 19112 leader_election.cc:288] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [CANDIDATE]: Term 1 election: Requested vote from peers e8915a8167a342c0b2e6355b33731b2d (127.8.43.129:43399), 7ece50ef90be40d6a5e9dc5393d70458 (127.8.43.131:44219)
I20260925 15:16:29.624413 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aff59d1728425aa1a948429635ccd1" candidate_uuid: "8c89f88cb91240e2878af5554580526c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8915a8167a342c0b2e6355b33731b2d"
I20260925 15:16:29.624439 19041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aff59d1728425aa1a948429635ccd1" candidate_uuid: "8c89f88cb91240e2878af5554580526c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ece50ef90be40d6a5e9dc5393d70458"
I20260925 15:16:29.624483 18773 raft_consensus.cc:3037] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:29.624497 19041 raft_consensus.cc:3037] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:29.625152 18773 raft_consensus.cc:2454] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c89f88cb91240e2878af5554580526c in term 1.
I20260925 15:16:29.625172 19041 raft_consensus.cc:2454] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c89f88cb91240e2878af5554580526c in term 1.
I20260925 15:16:29.625355 18841 leader_election.cc:302] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ece50ef90be40d6a5e9dc5393d70458, 8c89f88cb91240e2878af5554580526c; no voters:
I20260925 15:16:29.625453 19112 raft_consensus.cc:2781] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:29.625631 19112 raft_consensus.cc:686] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 1 LEADER]: Becoming Leader. State: Replica: 8c89f88cb91240e2878af5554580526c, State: Running, Role: LEADER
I20260925 15:16:29.625731 19112 consensus_queue.cc:237] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } }
I20260925 15:16:29.626564 18628 catalog_manager.cc:5495] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c89f88cb91240e2878af5554580526c (127.8.43.130). New cstate: current_term: 1 leader_uuid: "8c89f88cb91240e2878af5554580526c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c89f88cb91240e2878af5554580526c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260925 15:16:29.637965 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.635641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.8.43.130:40111)
0925 15:16:29.635702 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:29.635704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:29.635711 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:16:29.635739 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:29.635740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:29.635758 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:29.635830 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:29.635956 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.636441 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.637147 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:29.637364 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:29.637421 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:29.637575 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:29.637673 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:29.637740 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:29.637873 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 15:16:29.637894 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:16:29.639863 18773 raft_consensus.cc:1261] T 69aff59d1728425aa1a948429635ccd1 P e8915a8167a342c0b2e6355b33731b2d [term 1 FOLLOWER]: Refusing update from remote peer 8c89f88cb91240e2878af5554580526c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:29.639899 19041 raft_consensus.cc:1261] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 1 FOLLOWER]: Refusing update from remote peer 8c89f88cb91240e2878af5554580526c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:29.640254 19112 consensus_queue.cc:1035] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ece50ef90be40d6a5e9dc5393d70458" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:29.640450 19114 consensus_queue.cc:1035] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8915a8167a342c0b2e6355b33731b2d" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:29.643275 19119 mvcc.cc:205] Tried to move back new op lower bound from 7333271099961827328 to 7333271099907637248. Current Snapshot: MvccSnapshot[applied={T|T < 7333271099961827328}]
I20260925 15:16:29.643656 19118 mvcc.cc:205] Tried to move back new op lower bound from 7333271099961827328 to 7333271099907637248. Current Snapshot: MvccSnapshot[applied={T|T < 7333271099961827328 or (T in {7333271099961827328})}]
I20260925 15:16:29.645825 18627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:29.645918 18627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:29.646733 18627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3eab5ebcb0546b69be9273e72d1306c: Sending DeleteTablet for 3 replicas of tablet 69aff59d1728425aa1a948429635ccd1
I20260925 15:16:29.647099 18753 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aff59d1728425aa1a948429635ccd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:29 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:37788
I20260925 15:16:29.647127 19021 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aff59d1728425aa1a948429635ccd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:29 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:44040
I20260925 15:16:29.647159 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aff59d1728425aa1a948429635ccd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:29 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:52338
I20260925 15:16:29.647395 19128 tablet_replica.cc:330] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: stopping tablet replica
I20260925 15:16:29.647506 19128 raft_consensus.cc:2229] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:29.647639 19128 raft_consensus.cc:2258] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:29.647742 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 18690
I20260925 15:16:29.647802 19129 tablet_replica.cc:330] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: stopping tablet replica
I20260925 15:16:29.647909 19129 raft_consensus.cc:2229] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:29.648006 19129 raft_consensus.cc:2258] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:29.648911 19129 ts_tablet_manager.cc:1905] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:29.650213 19128 ts_tablet_manager.cc:1905] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:29.651106 19129 ts_tablet_manager.cc:1918] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:29.651188 19129 log.cc:1196] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/69aff59d1728425aa1a948429635ccd1
I20260925 15:16:29.651552 19129 ts_tablet_manager.cc:1939] T 69aff59d1728425aa1a948429635ccd1 P 7ece50ef90be40d6a5e9dc5393d70458: Deleting consensus metadata
I20260925 15:16:29.652093 18613 catalog_manager.cc:4841] TS 7ece50ef90be40d6a5e9dc5393d70458 (127.8.43.131:44219): tablet 69aff59d1728425aa1a948429635ccd1 (table test-table [id=513590aa2c4341a9ab92fc2239065713]) successfully deleted
I20260925 15:16:29.652524 19128 ts_tablet_manager.cc:1918] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:29.652658 19128 log.cc:1196] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/69aff59d1728425aa1a948429635ccd1
I20260925 15:16:29.652968 19128 ts_tablet_manager.cc:1939] T 69aff59d1728425aa1a948429635ccd1 P 8c89f88cb91240e2878af5554580526c: Deleting consensus metadata
I20260925 15:16:29.653369 18613 catalog_manager.cc:4841] TS 8c89f88cb91240e2878af5554580526c (127.8.43.130:40111): tablet 69aff59d1728425aa1a948429635ccd1 (table test-table [id=513590aa2c4341a9ab92fc2239065713]) successfully deleted
W20260925 15:16:29.655494 18615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:43399 (error 108)
I20260925 15:16:29.655766 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 18824
I20260925 15:16:29.656400 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:29.655904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:43399 (local address 127.0.0.1:37798)
0925 15:16:29.656318 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:29.656358 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
W20260925 15:16:29.656706 18615 catalog_manager.cc:4584] TS e8915a8167a342c0b2e6355b33731b2d (127.8.43.129:43399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69aff59d1728425aa1a948429635ccd1: Network error: Client connection negotiation failed: client connection to 127.8.43.129:43399: connect: Connection refused (error 111)
I20260925 15:16:29.660281 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 18958
I20260925 15:16:29.665184 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 18596
2026-09-25T15:16:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2161 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:16:29 dist-test-slave-c5d6 krb5kdc[19133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:16:29 dist-test-slave-c5d6 krb5kdc[19133](info): set up 2 sockets
Sep 25 15:16:29 dist-test-slave-c5d6 krb5kdc[19133](info): commencing operation
krb5kdc: starting...
W20260925 15:16:30.699553 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:16:30 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/rangeradmin_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/rangeradmin_127.8.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/rangerlookup_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/rangerlookup_127.8.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/HTTP_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/HTTP_127.8.43.148@KRBTEST.COM.keytab.
I20260925 15:16:30.852283 8366 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcJTRKe/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/postgres -l logfile start
2026-09-25 15:16:33.246 UTC [19163] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:16:33.246 UTC [19163] LOG: listening on IPv4 address "127.8.43.148", port 38311
2026-09-25 15:16:33.250 UTC [19163] LOG: listening on Unix socket "/tmp/.s.PGSQL.38311"
2026-09-25 15:16:33.285 UTC [19166] LOG: database system was shut down at 2026-09-25 15:16:31 UTC
2026-09-25 15:16:33.288 UTC [19163] LOG: database system is ready to accept connections
I20260925 15:16:34.227478 8366 mini_postgres.cc:96] Postgres bound to 38311
2026-09-25 15:16:34.233 UTC [19174] FATAL: database "slave" does not exist
127.8.43.148:38311 - accepting connections
I20260925 15:16:34.234040 8366 mini_ranger.cc:161] Starting Ranger...
I20260925 15:16:34.248176 8366 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:16:34.688381 8366 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:16:34.688470 8366 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-admin
2026-09-25 15:16:34,978 [I] DB FLAVOR :POSTGRES
2026-09-25 15:16:34,978 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:16:34,978 [I] Checking connection..
2026-09-25 15:16:34,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:16:35,236 [I] Checking connection passed.
2026-09-25 15:16:35,236 [I] --------- Verifying version history table ---------
2026-09-25 15:16:35,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:16:35,493 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:16:35,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:16:35,727 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:16:35,728 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:16:35,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:16:35.961 UTC [19238] WARNING: there is no transaction in progress
2026-09-25 15:16:35,967 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:16:35,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:16:36,216 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:16:36,217 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:16:36,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:16:36,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:16:36,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c5d6.c.gcp-upstream.internal','N') ;"
2026-09-25 15:16:36,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:16:37,184 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:16:37,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:16:37,434 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:16:37,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:16:37,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:16:37,918 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:16:37,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:16:38.848 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:38.864 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:38.880 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:38.894 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:38.909 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:38.926 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:38.989 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:38.997 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39.011 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39.022 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39.034 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39.048 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39.057 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39.063 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39.069 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39.075 UTC [19346] WARNING: there is no transaction in progress
2026-09-25 15:16:39,704 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:16:39,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:16:39,925 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:16:39,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:16:40,182 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:16:40,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:16:40,421 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:16:40,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:16:40,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:16:40,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c5d6.c.gcp-upstream.internal';"
2026-09-25 15:16:41,147 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:16:41,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:16:41,389 [I] DB_PATCHES have already been applied
I20260925 15:16:41.395339 8366 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcJTRKe/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:16:41.395406 8366 mini_ranger.cc:193] Using host: 127.8.43.148
I20260925 15:16:41.398310 8366 mini_ranger.cc:227] Ranger admin URL: http://127.8.43.148:50609
Sep 25, 2026 3:16:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:16:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:16:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:16:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:16:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:16:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:16:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:16:42.490609 8366 mini_ranger.cc:160] Time spent starting Ranger: real 8.257s user 0.004s sys 0.005s
Sep 25, 2026 3:17:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:17:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 3:17:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:17:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:17:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:17:04.218816 8366 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/rangerkms_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/rangerkms_127.8.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.43.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.43.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.43.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/HTTP_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.43.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/HTTP_127.8.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:17:04 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349424, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:17:04.296409 8366 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:17:04.305753 8366 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:17:05.148257 8366 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:17:05.148434 8366 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms
I20260925 15:17:05.148460 8366 mini_ranger_kms.cc:227] Using postgres at 127.8.43.148:38311
2026-09-25 15:17:05,329 [I] DB FLAVOR :POSTGRES
2026-09-25 15:17:05,330 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:17:05,330 [I] Checking connection
2026-09-25 15:17:05,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:17:05,583 [I] connection success
2026-09-25 15:17:05,583 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:17:05,583 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:17:05,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:17:05,842 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:17:05,842 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:17:05,843 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:17:05,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:38311/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:17:06,091 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:17:06.562577 8366 mini_ranger_kms.cc:310] Created kms service
I20260925 15:17:06.776062 8366 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:17:06.818204 8366 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:17:06 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349424, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:07.214789 8366 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:17:07.214908 8366 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:17:07.214929 8366 mini_ranger_kms.cc:240] Using host: 127.8.43.148
I20260925 15:17:07.218341 8366 mini_ranger_kms.cc:276] Ranger KMS PID: 19526
I20260925 15:17:07.218488 8366 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.43.148:37077
Sep 25, 2026 3:17:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:17:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:17:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:17:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:17:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:17:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:17:09.471433 8366 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.175s user 0.000s sys 0.163s
I20260925 15:17:09.471552 8366 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:17:09.471661 8366 mini_ranger_kms.cc:401] 127.8.43.148:37077/kms/v1/keys
Sep 25 15:17:09 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.43.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:17:09 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.43.148@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
Sep 25 15:17:10 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.43.148@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
Sep 25 15:17:12 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:17:12.499115 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:45673
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:45673
--ranger_config_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:37077/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:12.586462 19577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:12.586656 19577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:12.586678 19577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:12.587867 19577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:12.587914 19577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:12.587930 19577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:12.587944 19577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:12.588001 19577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:12.589639 19577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.43.148:37077/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:45673
--ranger_config_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:45673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:12.589891 19577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:12.590138 19577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:12.595294 19585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:12.595376 19577 server_base.cc:1034] running on GCE node
W20260925 15:17:12.595286 19583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:12.595484 19582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:12.595742 19577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:12.596068 19577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:12.597270 19577 hybrid_clock.cc:648] HybridClock initialized: now 1790349432597260 us; error 48 us; skew 500 ppm
Sep 25 15:17:12 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:12.603910 19577 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:17:12.604334 19577 webserver.cc:460] Webserver started at http://127.8.43.190:45855/ using document root <none> and password file <none>
I20260925 15:17:12.604552 19577 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:12.604612 19577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:12.604753 19577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:17:12 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:12.651306 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "80845c9b683d4ab89cec8b9f369ee56e"
format_stamp: "Formatted at 2026-09-25 15:17:12 on dist-test-slave-c5d6"
server_key: "f9873e1d113ddad820ecd70260f9f0e1"
server_key_iv: "cb32cec8a85570a6d25ce7943e3fd3ab"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:12.651726 19577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "80845c9b683d4ab89cec8b9f369ee56e"
format_stamp: "Formatted at 2026-09-25 15:17:12 on dist-test-slave-c5d6"
server_key: "f9873e1d113ddad820ecd70260f9f0e1"
server_key_iv: "cb32cec8a85570a6d25ce7943e3fd3ab"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:12.653177 19577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:17:12.664682 19592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:12.664899 19577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:17:12.665019 19577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "80845c9b683d4ab89cec8b9f369ee56e"
format_stamp: "Formatted at 2026-09-25 15:17:12 on dist-test-slave-c5d6"
server_key: "f9873e1d113ddad820ecd70260f9f0e1"
server_key_iv: "cb32cec8a85570a6d25ce7943e3fd3ab"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:12.665177 19577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:12.686358 19577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:12.688203 19577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:12.688351 19577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:12.692654 19577 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:45673
I20260925 15:17:12.692708 19644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:45673 every 8 connection(s)
I20260925 15:17:12.693111 19577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:12.693763 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:12.694841 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 19577
I20260925 15:17:12.694947 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:17:12.695134 8366 external_mini_cluster.cc:1428] Setting key d3ad14373b17f0f20ac6fd284ad3dacb
I20260925 15:17:12.696583 19645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e: Bootstrap starting.
I20260925 15:17:12.697505 19645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:12.697853 19645 log.cc:824] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:12.698556 19645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e: No bootstrap required, opened a new log
I20260925 15:17:12.700824 19645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80845c9b683d4ab89cec8b9f369ee56e" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45673 } }
I20260925 15:17:12.700956 19645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:12.700979 19645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80845c9b683d4ab89cec8b9f369ee56e, State: Initialized, Role: FOLLOWER
I20260925 15:17:12.701128 19645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80845c9b683d4ab89cec8b9f369ee56e" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45673 } }
I20260925 15:17:12.701205 19645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:12.701228 19645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:12.701296 19645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:12.701807 19645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80845c9b683d4ab89cec8b9f369ee56e" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45673 } }
I20260925 15:17:12.701936 19645 leader_election.cc:302] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80845c9b683d4ab89cec8b9f369ee56e; no voters:
I20260925 15:17:12.702126 19645 leader_election.cc:288] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:12.702149 19650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:12.702276 19650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [term 1 LEADER]: Becoming Leader. State: Replica: 80845c9b683d4ab89cec8b9f369ee56e, State: Running, Role: LEADER
I20260925 15:17:12.702404 19645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:12.702420 19650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80845c9b683d4ab89cec8b9f369ee56e" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45673 } }
Sep 25 15:17:12 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:12.703152 19651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80845c9b683d4ab89cec8b9f369ee56e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80845c9b683d4ab89cec8b9f369ee56e" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45673 } } }
I20260925 15:17:12.703265 19651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:12.703828 19652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80845c9b683d4ab89cec8b9f369ee56e. Latest consensus state: current_term: 1 leader_uuid: "80845c9b683d4ab89cec8b9f369ee56e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80845c9b683d4ab89cec8b9f369ee56e" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 45673 } } }
I20260925 15:17:12.703922 19652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:12.704053 19660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:12.704151 19645 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:17:12.704548 19660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:12.710935 19660 catalog_manager.cc:1347] Generated new cluster ID: fde61cbdd945408f91735318ff2622bb
I20260925 15:17:12.711001 19660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:12.711098 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:12.695696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.8.43.190:45673)
0925 15:17:12.695874 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:12.695902 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:12.695935 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:17:12.696461 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:12.696469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:12.696526 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:12.696688 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:12.697297 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:12.698738 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:12.699564 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:12.700231 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:12.700348 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:12.703108 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:12.703113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:12.703114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:12.703126 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:12.707860 (+ 4734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:12.708404 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:12.708426 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:12.708426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:12.708577 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:12.708809 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:12.708810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:12.708811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:12.709422 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:12.710491 (+ 1069us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:12.710591 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:17:12.710791 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260925 15:17:12.745489 19660 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:12.746253 19660 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:12.752216 19660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e: Generated new TSK 0
I20260925 15:17:12.752444 19660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:17:13.433706 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:37077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45673
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:13.518904 19690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:13.519138 19690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:13.519177 19690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:13.520340 19690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:13.520407 19690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:13.520449 19690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:17:13.521916 19690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.43.148:37077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:13.522171 19690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:13.522392 19690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:13.528376 19695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:13.528434 19696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:13.528589 19690 server_base.cc:1034] running on GCE node
W20260925 15:17:13.528815 19698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:13.529062 19690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:13.529366 19690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:13.530529 19690 hybrid_clock.cc:648] HybridClock initialized: now 1790349433530519 us; error 30 us; skew 500 ppm
Sep 25 15:17:13 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:13.533131 19690 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:17:13.533600 19690 webserver.cc:460] Webserver started at http://127.8.43.129:42081/ using document root <none> and password file <none>
I20260925 15:17:13.533816 19690 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:13.533879 19690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:13.534006 19690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:17:13 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:13.548880 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "7fdb767b1c844cf7ba9aa073d123836f"
format_stamp: "Formatted at 2026-09-25 15:17:13 on dist-test-slave-c5d6"
server_key: "767f4d36746bd79f4f2c86445af10c9e"
server_key_iv: "e677683d003eb8a8dc7e271179fe30ed"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:13.549466 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "7fdb767b1c844cf7ba9aa073d123836f"
format_stamp: "Formatted at 2026-09-25 15:17:13 on dist-test-slave-c5d6"
server_key: "767f4d36746bd79f4f2c86445af10c9e"
server_key_iv: "e677683d003eb8a8dc7e271179fe30ed"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:13.551337 19690 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:17:13.568121 19705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:13.568344 19690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:17:13.568480 19690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "7fdb767b1c844cf7ba9aa073d123836f"
format_stamp: "Formatted at 2026-09-25 15:17:13 on dist-test-slave-c5d6"
server_key: "767f4d36746bd79f4f2c86445af10c9e"
server_key_iv: "e677683d003eb8a8dc7e271179fe30ed"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:13.568600 19690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:13.602056 19690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:13.604485 19690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:13.604642 19690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:13.604867 19690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:13.605183 19690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:13.605237 19690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:13.605278 19690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:13.605304 19690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:13.612512 19690 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:35357
I20260925 15:17:13.612859 19690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:13.613466 19820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:35357 every 8 connection(s)
I20260925 15:17:13.619166 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 19690
I20260925 15:17:13.619297 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:17:13.619517 8366 external_mini_cluster.cc:1428] Setting key 5c55671c5e41fdb56506ac6e70db26b4
Sep 25 15:17:13 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:13.624101 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:13.613594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:45673 (local address 127.8.43.129:59311)
0925 15:17:13.613733 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:13.613772 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:13.614244 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:13.615550 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:13.615555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:13.615812 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:13.616027 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:13.616032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:13.616748 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:13.616749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:13.617746 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:13.617749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:13.618257 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:13.618258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:13.618396 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:13.618959 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:13.618969 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:13.620614 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:13.622430 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:13.622437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:13.622438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:13.622662 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:13.622942 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:13.622943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:13.622944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:13.623079 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:13.623402 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:13.623408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:13.623581 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 15:17:13.623693 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:17:13.623853 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260925 15:17:13.624524 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:13.613637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:59311 (local address 127.8.43.190:45673)
0925 15:17:13.615342 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:13.615344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:13.615354 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:17:13.615382 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:13.615382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:13.615396 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:13.615468 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:13.616136 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:13.616661 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:13.617863 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:13.618129 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:13.618191 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:13.620715 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:13.620717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:13.620718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:13.620726 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:13.622331 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:13.622791 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:13.622792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:13.622793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:13.622862 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:13.623151 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:13.623152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:13.623152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:13.623316 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:13.623375 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:13.624366 (+ 991us) server_negotiation.cc:299] Negotiation successful
0925 15:17:13.624446 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1669,"thread_start_us":92,"threads_started":1}
I20260925 15:17:13.624861 19821 heartbeater.cc:347] Connected to a master server at 127.8.43.190:45673
I20260925 15:17:13.624970 19821 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:13.625161 19821 heartbeater.cc:511] Master 127.8.43.190:45673 requested a full tablet report, sending...
I20260925 15:17:13.625659 19609 ts_manager.cc:194] Registered new tserver with Master: 7fdb767b1c844cf7ba9aa073d123836f (127.8.43.129:35357)
I20260925 15:17:13.626288 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:59311
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:17:13.654448 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:37077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45673
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:17:13 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:17:13.784719 19827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:13.785001 19827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:13.785037 19827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:13.786813 19827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:13.786878 19827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:13.786918 19827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:17:13.789112 19827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.43.148:37077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:13.789359 19827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:13.789630 19827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:13.795639 19835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:13.795636 19834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:13.795807 19827 server_base.cc:1034] running on GCE node
W20260925 15:17:13.796003 19837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:13.796226 19827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:13.796479 19827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:13.797652 19827 hybrid_clock.cc:648] HybridClock initialized: now 1790349433797632 us; error 31 us; skew 500 ppm
Sep 25 15:17:13 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:13.800809 19827 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:17:13.801277 19827 webserver.cc:460] Webserver started at http://127.8.43.130:38557/ using document root <none> and password file <none>
I20260925 15:17:13.801533 19827 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:13.801587 19827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:13.801719 19827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:17:13 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:13.815899 19827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "f6994b7dc799431590a5a022b07f378b"
format_stamp: "Formatted at 2026-09-25 15:17:13 on dist-test-slave-c5d6"
server_key: "b58d8995971e92c707c93a64cfd5087e"
server_key_iv: "40eb87cb028cbac0ebe2592ad0412d47"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:13.816329 19827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "f6994b7dc799431590a5a022b07f378b"
format_stamp: "Formatted at 2026-09-25 15:17:13 on dist-test-slave-c5d6"
server_key: "b58d8995971e92c707c93a64cfd5087e"
server_key_iv: "40eb87cb028cbac0ebe2592ad0412d47"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:13.817899 19827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:17:13.835372 19845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:13.835567 19827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260925 15:17:13.835655 19827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "f6994b7dc799431590a5a022b07f378b"
format_stamp: "Formatted at 2026-09-25 15:17:13 on dist-test-slave-c5d6"
server_key: "b58d8995971e92c707c93a64cfd5087e"
server_key_iv: "40eb87cb028cbac0ebe2592ad0412d47"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:13.835778 19827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:13.854847 19827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:13.857288 19827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:13.857461 19827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:13.857688 19827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:13.858081 19827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:13.858135 19827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:13.858171 19827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:13.858198 19827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:13.864941 19827 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:46755
I20260925 15:17:13.865384 19827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:17:13.868728 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 19827
I20260925 15:17:13.868853 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:17:13.869043 8366 external_mini_cluster.cc:1428] Setting key 9fa7a3bfbd34b8ed2de3104ee5ff2254
I20260925 15:17:13.866048 19958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:46755 every 8 connection(s)
Sep 25 15:17:13 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:13.877255 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:13.866432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:55493 (local address 127.8.43.190:45673)
0925 15:17:13.866476 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:13.866478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:13.866490 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:17:13.867171 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:13.867173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:13.867190 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:13.867259 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:13.868157 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:13.868931 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:13.870798 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:13.871113 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:13.871160 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:13.873723 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:13.873725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:13.873725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:13.873734 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:13.875542 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:13.876148 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:13.876149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:13.876149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:13.876213 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:13.876494 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:13.876495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:13.876495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:13.876636 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:13.876678 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:13.877073 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 15:17:13.877159 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:17:13.879624 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:13.866147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:45673 (local address 127.8.43.130:55493)
0925 15:17:13.866365 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:13.866408 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:13.867026 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:13.867481 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:13.867486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:13.867758 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:13.868034 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:13.868040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:13.869071 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:13.869072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:13.870698 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:13.870701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:13.871276 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:13.871277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:13.871437 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:13.872020 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:13.872030 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:13.873610 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:13.875646 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:13.875653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:13.875655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:13.875985 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:13.876289 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:13.876289 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:13.876290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:13.876423 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:13.876724 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:13.876730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:13.876904 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:17:13.879183 (+ 2279us) client_negotiation.cc:240] Negotiation successful
0925 15:17:13.879374 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260925 15:17:13.886588 19959 heartbeater.cc:347] Connected to a master server at 127.8.43.190:45673
I20260925 15:17:13.886709 19959 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:13.886924 19959 heartbeater.cc:511] Master 127.8.43.190:45673 requested a full tablet report, sending...
I20260925 15:17:13.887359 19609 ts_manager.cc:194] Registered new tserver with Master: f6994b7dc799431590a5a022b07f378b (127.8.43.130:46755)
I20260925 15:17:13.887859 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:55493
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:17:13.917191 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:37077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45673
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:14.021279 19964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:14.021513 19964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:14.021550 19964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:14.022786 19964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:14.022847 19964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:14.022892 19964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:17:14.024492 19964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.43.148:37077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:45673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:14.024735 19964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:14.025032 19964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:14.030521 19969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:14.030610 19964 server_base.cc:1034] running on GCE node
W20260925 15:17:14.030556 19970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:14.030781 19972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:14.030988 19964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:14.031301 19964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:14.032454 19964 hybrid_clock.cc:648] HybridClock initialized: now 1790349434032446 us; error 22 us; skew 500 ppm
Sep 25 15:17:14 dist-test-slave-c5d6 krb5kdc[19133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:14.034883 19964 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:17:14.035363 19964 webserver.cc:460] Webserver started at http://127.8.43.131:39741/ using document root <none> and password file <none>
I20260925 15:17:14.035593 19964 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:14.035665 19964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:14.035801 19964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:17:14 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:14.048473 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "6221b11b916f42ea9f8be5a54efadba0"
format_stamp: "Formatted at 2026-09-25 15:17:14 on dist-test-slave-c5d6"
server_key: "c6ed87dcac6c5f6c27ec1c1f628b1fce"
server_key_iv: "919bd83e503c52a8b87e9065dda975d3"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:14.048955 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "6221b11b916f42ea9f8be5a54efadba0"
format_stamp: "Formatted at 2026-09-25 15:17:14 on dist-test-slave-c5d6"
server_key: "c6ed87dcac6c5f6c27ec1c1f628b1fce"
server_key_iv: "919bd83e503c52a8b87e9065dda975d3"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:14.053465 19964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 15:17:14.075973 19979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:14.076184 19964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 15:17:14.076293 19964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "6221b11b916f42ea9f8be5a54efadba0"
format_stamp: "Formatted at 2026-09-25 15:17:14 on dist-test-slave-c5d6"
server_key: "c6ed87dcac6c5f6c27ec1c1f628b1fce"
server_key_iv: "919bd83e503c52a8b87e9065dda975d3"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:14.076409 19964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:14.099015 19964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:14.101311 19964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:14.101513 19964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:14.101779 19964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:14.102151 19964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:14.102207 19964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:14.102244 19964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:14.102265 19964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:14.111945 19964 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:44895
I20260925 15:17:14.112372 19964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:14.113636 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 19964
I20260925 15:17:14.113734 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:17:14.113914 8366 external_mini_cluster.cc:1428] Setting key ecc7adf6864675460dc6363548a135e4
I20260925 15:17:14.113046 20092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:44895 every 8 connection(s)
Sep 25 15:17:14 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:14.125735 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.113805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:37979 (local address 127.8.43.190:45673)
0925 15:17:14.114137 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:14.114140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:14.115164 (+ 1024us) server_negotiation.cc:407] Connection header received
0925 15:17:14.115878 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:14.115879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:14.115891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:14.115922 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:14.116881 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.117662 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.119179 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.119493 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.119538 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.122155 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:14.122157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:14.122157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:14.122165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:14.123964 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:14.124447 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:14.124448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:14.124448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:14.124506 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:14.124802 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:14.124803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:14.124804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:14.124943 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:14.124990 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:14.125536 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 15:17:14.125631 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260925 15:17:14.126446 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.113679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:45673 (local address 127.8.43.131:37979)
0925 15:17:14.115008 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:14.115053 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:14.115756 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:14.116124 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:14.116129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:14.116476 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:14.116767 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.116774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.117800 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.117801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.119080 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.119082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.119646 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.119647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.119776 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.120400 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:14.120410 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:14.122045 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:14.124055 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:14.124061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:14.124063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:14.124349 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:14.124581 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:14.124582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:14.124582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:14.124719 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:14.125039 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:14.125044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:14.125198 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 15:17:14.126057 (+ 859us) client_negotiation.cc:240] Negotiation successful
0925 15:17:14.126223 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1288,"thread_start_us":619,"threads_started":1}
I20260925 15:17:14.127199 20093 heartbeater.cc:347] Connected to a master server at 127.8.43.190:45673
I20260925 15:17:14.127328 20093 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:14.127502 20093 heartbeater.cc:511] Master 127.8.43.190:45673 requested a full tablet report, sending...
I20260925 15:17:14.127938 19609 ts_manager.cc:194] Registered new tserver with Master: 6221b11b916f42ea9f8be5a54efadba0 (127.8.43.131:44895)
I20260925 15:17:14.128443 19609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:37979
I20260925 15:17:14.136757 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:17:14.147110 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.138635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.8.43.190:45673)
0925 15:17:14.138682 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:14.138684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:14.138831 (+ 147us) server_negotiation.cc:407] Connection header received
0925 15:17:14.138866 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:14.138867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:14.138877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:14.138942 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:14.139652 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.140495 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.141510 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.141713 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.141796 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.143072 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:14.143073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:14.143074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:14.143081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:14.144592 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:14.144956 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:14.144957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:14.144958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:14.145014 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:14.145217 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:14.145218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:14.145218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:14.145326 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:14.145440 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:14.146914 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0925 15:17:14.146999 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:17:14.148990 19609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:17:14.151150 19609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:17:14.161818 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.155447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:35357 (local address 127.0.0.1:39556)
0925 15:17:14.156496 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:14.156506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:14.156557 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:14.156827 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:14.156827 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:14.156829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:14.156881 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.156882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.158963 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.158964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.160019 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.160022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.161464 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.161465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.161559 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.161576 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 15:17:14.161659 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:17:14.161720 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":72,"threads_started":1}
I20260925 15:17:14.162299 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.155658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.8.43.129:35357)
0925 15:17:14.156166 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:14.156171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:14.156623 (+ 452us) server_negotiation.cc:407] Connection header received
0925 15:17:14.156660 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:14.156662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:14.156679 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:14.156757 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:14.156985 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.158866 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.160145 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.161242 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.161324 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.161389 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:14.162164 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 15:17:14.162212 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":61,"threads_started":1}
I20260925 15:17:14.163327 19737 tablet_service.cc:1467] Processing CreateTablet for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7 (DEFAULT_TABLE table=test-table [id=c67fd9f9e6784af988cee1b0c2bae466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:14.163728 19737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:14.165807 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.154972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:44895 (local address 127.0.0.1:34550)
0925 15:17:14.155878 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:14.155888 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:14.155931 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:14.156416 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:14.156419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:14.156422 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:14.156498 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.156501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.158056 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.158057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.159390 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.159393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.165523 (+ 6130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.165525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.165616 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.165621 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:17:14.165689 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:17:14.165731 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":72,"threads_started":1}
I20260925 15:17:14.166075 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.155191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.8.43.131:44895)
0925 15:17:14.156214 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:14.156217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:14.156226 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:17:14.156259 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:14.156260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:14.156274 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:14.156390 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:14.156606 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.157960 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.159523 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.164411 (+ 4888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.164492 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.164578 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:14.165931 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0925 15:17:14.165981 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":85,"threads_started":1}
I20260925 15:17:14.166947 20027 tablet_service.cc:1467] Processing CreateTablet for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7 (DEFAULT_TABLE table=test-table [id=c67fd9f9e6784af988cee1b0c2bae466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:14.167315 20027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:14.169303 20112 tablet_bootstrap.cc:492] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f: Bootstrap starting.
I20260925 15:17:14.170406 20112 tablet_bootstrap.cc:654] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:14.170847 20112 log.cc:824] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:14.171561 20113 tablet_bootstrap.cc:492] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: Bootstrap starting.
I20260925 15:17:14.172607 20113 tablet_bootstrap.cc:654] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:14.174443 20113 log.cc:824] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:14.175203 20113 tablet_bootstrap.cc:492] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: No bootstrap required, opened a new log
I20260925 15:17:14.175323 20113 ts_tablet_manager.cc:1397] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:17:14.176025 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.154741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:46755 (local address 127.0.0.1:45122)
0925 15:17:14.155588 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:14.155630 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:14.155700 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:14.171069 (+ 15369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:14.171070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:14.171072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:14.171124 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.171126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.172957 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.172958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.174481 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.174484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.175770 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.175771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.175832 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.175837 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:17:14.175891 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:17:14.175933 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":88,"threads_started":1}
I20260925 15:17:14.176535 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.154840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.8.43.130:46755)
0925 15:17:14.168889 (+ 14049us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:14.168895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:14.168909 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:17:14.168951 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:14.168953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:14.168970 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:14.169078 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:14.171207 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.172853 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.174664 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.175593 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.175672 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.175733 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:14.176385 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 15:17:14.176447 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14001,"thread_start_us":75,"threads_started":1}
I20260925 15:17:14.177371 19893 tablet_service.cc:1467] Processing CreateTablet for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7 (DEFAULT_TABLE table=test-table [id=c67fd9f9e6784af988cee1b0c2bae466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:14.177610 20112 tablet_bootstrap.cc:492] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f: No bootstrap required, opened a new log
I20260925 15:17:14.177682 20112 ts_tablet_manager.cc:1397] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 15:17:14.177712 19893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:14.177632 20113 raft_consensus.cc:348] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.177824 20113 raft_consensus.cc:374] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:14.177863 20113 raft_consensus.cc:729] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6221b11b916f42ea9f8be5a54efadba0, State: Initialized, Role: FOLLOWER
I20260925 15:17:14.178006 20113 consensus_queue.cc:260] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.178296 20113 ts_tablet_manager.cc:1428] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:17:14.178431 20093 heartbeater.cc:503] Master 127.8.43.190:45673 was elected leader, sending a full tablet report...
I20260925 15:17:14.180533 20112 raft_consensus.cc:348] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.180692 20112 raft_consensus.cc:374] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:14.180732 20112 raft_consensus.cc:729] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fdb767b1c844cf7ba9aa073d123836f, State: Initialized, Role: FOLLOWER
I20260925 15:17:14.180945 20117 tablet_bootstrap.cc:492] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: Bootstrap starting.
I20260925 15:17:14.181527 20112 consensus_queue.cc:260] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.181885 20112 ts_tablet_manager.cc:1428] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:17:14.181936 20117 tablet_bootstrap.cc:654] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:14.182101 19821 heartbeater.cc:503] Master 127.8.43.190:45673 was elected leader, sending a full tablet report...
I20260925 15:17:14.182238 20117 log.cc:824] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:14.183259 20117 tablet_bootstrap.cc:492] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: No bootstrap required, opened a new log
I20260925 15:17:14.183332 20117 ts_tablet_manager.cc:1397] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:17:14.184670 20117 raft_consensus.cc:348] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.184819 20117 raft_consensus.cc:374] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:14.184866 20117 raft_consensus.cc:729] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6994b7dc799431590a5a022b07f378b, State: Initialized, Role: FOLLOWER
I20260925 15:17:14.184988 20117 consensus_queue.cc:260] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.185278 20117 ts_tablet_manager.cc:1428] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:17:14.185302 19959 heartbeater.cc:503] Master 127.8.43.190:45673 was elected leader, sending a full tablet report...
Sep 25 15:17:14 dist-test-slave-c5d6 krb5kdc[19133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:14.241616 20116 raft_consensus.cc:482] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:17:14.241756 20116 raft_consensus.cc:504] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.242195 20116 leader_election.cc:288] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fdb767b1c844cf7ba9aa073d123836f (127.8.43.129:35357), f6994b7dc799431590a5a022b07f378b (127.8.43.130:46755)
I20260925 15:17:14.246059 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.242368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:35357 (local address 127.8.43.131:49885)
0925 15:17:14.242473 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:14.242484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:14.242524 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:14.242906 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:14.242906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:14.242908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:14.242974 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.242975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.243893 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.243894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.245061 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.245063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.245812 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.245813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.245886 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.245890 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:17:14.245950 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:17:14.245985 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 15:17:14.246075 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.242605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:49885 (local address 127.8.43.129:35357)
0925 15:17:14.242759 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:14.242762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:14.242771 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:17:14.242802 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:14.242803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:14.242817 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:14.242881 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:14.243054 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.243798 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.245181 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.245725 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.245795 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.245804 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:14.245969 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 15:17:14.246009 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 15:17:14.246448 19763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d236d40c8fb4c759bdf9c38c3da4bc7" candidate_uuid: "6221b11b916f42ea9f8be5a54efadba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fdb767b1c844cf7ba9aa073d123836f" is_pre_election: true
I20260925 15:17:14.246634 19763 raft_consensus.cc:2454] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6221b11b916f42ea9f8be5a54efadba0 in term 0.
I20260925 15:17:14.246889 19983 leader_election.cc:302] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6221b11b916f42ea9f8be5a54efadba0, 7fdb767b1c844cf7ba9aa073d123836f; no voters:
I20260925 15:17:14.247028 20116 raft_consensus.cc:2781] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:17:14.247083 20116 raft_consensus.cc:482] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:17:14.247117 20116 raft_consensus.cc:3037] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:14.248057 20116 raft_consensus.cc:504] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.248234 20116 leader_election.cc:288] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [CANDIDATE]: Term 1 election: Requested vote from peers 7fdb767b1c844cf7ba9aa073d123836f (127.8.43.129:35357), f6994b7dc799431590a5a022b07f378b (127.8.43.130:46755)
I20260925 15:17:14.248413 19763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d236d40c8fb4c759bdf9c38c3da4bc7" candidate_uuid: "6221b11b916f42ea9f8be5a54efadba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fdb767b1c844cf7ba9aa073d123836f"
I20260925 15:17:14.248499 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.242942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:37121 (local address 127.8.43.130:46755)
0925 15:17:14.242988 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:14.242990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:14.242997 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:17:14.243020 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:14.243021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:14.243029 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:14.243076 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:14.244528 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.245243 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.247526 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.248084 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.248145 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.248157 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:14.248394 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:17:14.248441 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:17:14.248493 19763 raft_consensus.cc:3037] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:14.248723 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.242440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:46755 (local address 127.8.43.131:37121)
0925 15:17:14.242672 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:14.242680 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:14.242720 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:14.244309 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:14.244310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:14.244313 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:14.244400 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.244401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.245368 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.245369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.247412 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:14.247414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.248233 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.248234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:14.248308 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.248311 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:17:14.248577 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 15:17:14.248617 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
I20260925 15:17:14.249032 19910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d236d40c8fb4c759bdf9c38c3da4bc7" candidate_uuid: "6221b11b916f42ea9f8be5a54efadba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6994b7dc799431590a5a022b07f378b" is_pre_election: true
I20260925 15:17:14.249172 19910 raft_consensus.cc:2454] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6221b11b916f42ea9f8be5a54efadba0 in term 0.
I20260925 15:17:14.249293 19903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d236d40c8fb4c759bdf9c38c3da4bc7" candidate_uuid: "6221b11b916f42ea9f8be5a54efadba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6994b7dc799431590a5a022b07f378b"
I20260925 15:17:14.249361 19903 raft_consensus.cc:3037] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:14.249387 19763 raft_consensus.cc:2454] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6221b11b916f42ea9f8be5a54efadba0 in term 1.
I20260925 15:17:14.249550 19983 leader_election.cc:302] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6221b11b916f42ea9f8be5a54efadba0, 7fdb767b1c844cf7ba9aa073d123836f; no voters:
I20260925 15:17:14.249656 20116 raft_consensus.cc:2781] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:14.249833 20116 raft_consensus.cc:686] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 1 LEADER]: Becoming Leader. State: Replica: 6221b11b916f42ea9f8be5a54efadba0, State: Running, Role: LEADER
I20260925 15:17:14.249931 20116 consensus_queue.cc:237] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } }
I20260925 15:17:14.250248 19903 raft_consensus.cc:2454] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6221b11b916f42ea9f8be5a54efadba0 in term 1.
I20260925 15:17:14.250716 19609 catalog_manager.cc:5495] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6221b11b916f42ea9f8be5a54efadba0 (127.8.43.131). New cstate: current_term: 1 leader_uuid: "6221b11b916f42ea9f8be5a54efadba0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6221b11b916f42ea9f8be5a54efadba0" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 44895 } health_report { overall_health: HEALTHY } } }
I20260925 15:17:14.279327 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.276152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.8.43.131:44895)
0925 15:17:14.276207 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:14.276209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:14.276220 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:17:14.276261 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:14.276262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:14.276285 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:14.276373 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:17:14.276543 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.277250 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.278383 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:14.278658 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:14.278721 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:14.278932 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:17:14.279071 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:17:14.279165 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:14.279196 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 15:17:14.279239 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:17:14.281422 19903 raft_consensus.cc:1261] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 1 FOLLOWER]: Refusing update from remote peer 6221b11b916f42ea9f8be5a54efadba0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:14.281510 19763 raft_consensus.cc:1261] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 7fdb767b1c844cf7ba9aa073d123836f [term 1 FOLLOWER]: Refusing update from remote peer 6221b11b916f42ea9f8be5a54efadba0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:14.281749 20116 consensus_queue.cc:1035] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6994b7dc799431590a5a022b07f378b" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:14.282085 20116 consensus_queue.cc:1035] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fdb767b1c844cf7ba9aa073d123836f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:14.285017 20126 mvcc.cc:205] Tried to move back new op lower bound from 7333271282814070784 to 7333271282688319488. Current Snapshot: MvccSnapshot[applied={T|T < 7333271282814070784}]
I20260925 15:17:14.285754 20125 mvcc.cc:205] Tried to move back new op lower bound from 7333271282814070784 to 7333271282688319488. Current Snapshot: MvccSnapshot[applied={T|T < 7333271282814070784 or (T in {7333271282814070784})}]
I20260925 15:17:14.287333 19609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:14.287425 19609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:14.288085 19609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80845c9b683d4ab89cec8b9f369ee56e: Sending DeleteTablet for 3 replicas of tablet 8d236d40c8fb4c759bdf9c38c3da4bc7
I20260925 15:17:14.288475 19893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:14 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:45122
I20260925 15:17:14.288666 20027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:14 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:34550
I20260925 15:17:14.288960 19737 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:14 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:39556
I20260925 15:17:14.289430 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 19690
I20260925 15:17:14.295392 20133 tablet_replica.cc:330] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: stopping tablet replica
I20260925 15:17:14.295557 20133 raft_consensus.cc:2229] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:14.295717 20133 raft_consensus.cc:2258] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:14.296206 20133 ts_tablet_manager.cc:1905] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:14.297897 20134 tablet_replica.cc:330] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: stopping tablet replica
I20260925 15:17:14.298054 20134 raft_consensus.cc:2229] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:17:14.298235 20134 raft_consensus.cc:2258] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 15:17:14.302175 19983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:35357 (error 108)
I20260925 15:17:14.302659 20133 ts_tablet_manager.cc:1918] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:14.303097 20134 ts_tablet_manager.cc:1905] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:14.303264 20133 log.cc:1196] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/8d236d40c8fb4c759bdf9c38c3da4bc7
I20260925 15:17:14.304162 20133 ts_tablet_manager.cc:1939] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P f6994b7dc799431590a5a022b07f378b: Deleting consensus metadata
I20260925 15:17:14.305974 19595 catalog_manager.cc:4841] TS f6994b7dc799431590a5a022b07f378b (127.8.43.130:46755): tablet 8d236d40c8fb4c759bdf9c38c3da4bc7 (table test-table [id=c67fd9f9e6784af988cee1b0c2bae466]) successfully deleted
I20260925 15:17:14.306213 20134 ts_tablet_manager.cc:1918] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:14.306315 20134 log.cc:1196] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/8d236d40c8fb4c759bdf9c38c3da4bc7
I20260925 15:17:14.306600 20134 ts_tablet_manager.cc:1939] T 8d236d40c8fb4c759bdf9c38c3da4bc7 P 6221b11b916f42ea9f8be5a54efadba0: Deleting consensus metadata
I20260925 15:17:14.306991 19595 catalog_manager.cc:4841] TS 6221b11b916f42ea9f8be5a54efadba0 (127.8.43.131:44895): tablet 8d236d40c8fb4c759bdf9c38c3da4bc7 (table test-table [id=c67fd9f9e6784af988cee1b0c2bae466]) successfully deleted
W20260925 15:17:14.307478 19596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:35357 (error 108)
W20260925 15:17:14.307901 20121 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:17:14.307338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:35357 (local address 127.8.43.131:52161)
0925 15:17:14.307383 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:14.307396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:14.307450 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:14.307860 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:35357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 15:17:14.308627 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 19827
I20260925 15:17:14.308642 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:14.308502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:35357 (local address 127.0.0.1:39562)
0925 15:17:14.308552 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:14.308611 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:17:14.310673 19596 catalog_manager.cc:4584] TS 7fdb767b1c844cf7ba9aa073d123836f (127.8.43.129:35357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d236d40c8fb4c759bdf9c38c3da4bc7: Network error: Client connection negotiation failed: client connection to 127.8.43.129:35357: connect: Connection refused (error 111)
I20260925 15:17:14.314522 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 19964
I20260925 15:17:14.320960 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 19577
I20260925 15:17:14.327896 8366 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:17:14.671823 8366 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:17:14.671928 8366 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:17:15.016295 8366 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:17:15.016 UTC [19163] LOG: received smart shutdown request
2026-09-25 15:17:15.020 UTC [19163] LOG: background worker "logical replication launcher" (PID 19172) exited with exit code 1
2026-09-25 15:17:15.020 UTC [19167] LOG: shutting down
2026-09-25 15:17:15.187 UTC [19163] LOG: database system is shut down
2026-09-25T15:17:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45624 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:17:15 dist-test-slave-c5d6 krb5kdc[20146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:17:15 dist-test-slave-c5d6 krb5kdc[20146](info): set up 2 sockets
Sep 25 15:17:15 dist-test-slave-c5d6 krb5kdc[20146](info): commencing operation
krb5kdc: starting...
W20260925 15:17:16.343303 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:17:16 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/rangeradmin_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/rangeradmin_127.8.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/rangerlookup_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/rangerlookup_127.8.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/HTTP_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/HTTP_127.8.43.148@KRBTEST.COM.keytab.
I20260925 15:17:16.505393 8366 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcJTRKe/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/postgres -l logfile start
2026-09-25 15:17:19.069 UTC [20176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:17:19.069 UTC [20176] LOG: listening on IPv4 address "127.8.43.148", port 60333
2026-09-25 15:17:19.073 UTC [20176] LOG: listening on Unix socket "/tmp/.s.PGSQL.60333"
2026-09-25 15:17:19.108 UTC [20179] LOG: database system was shut down at 2026-09-25 15:17:17 UTC
2026-09-25 15:17:19.113 UTC [20176] LOG: database system is ready to accept connections
I20260925 15:17:20.059253 8366 mini_postgres.cc:96] Postgres bound to 60333
2026-09-25 15:17:20.065 UTC [20187] FATAL: database "slave" does not exist
127.8.43.148:60333 - accepting connections
I20260925 15:17:20.066103 8366 mini_ranger.cc:161] Starting Ranger...
I20260925 15:17:20.080858 8366 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:17:20.565304 8366 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:17:20.565393 8366 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-admin
2026-09-25 15:17:20,841 [I] DB FLAVOR :POSTGRES
2026-09-25 15:17:20,841 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:17:20,841 [I] Checking connection..
2026-09-25 15:17:20,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:17:21,079 [I] Checking connection passed.
2026-09-25 15:17:21,079 [I] --------- Verifying version history table ---------
2026-09-25 15:17:21,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:17:21,321 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:17:21,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:17:21,571 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:17:21,571 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:17:21,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:17:21.801 UTC [20251] WARNING: there is no transaction in progress
2026-09-25 15:17:21,807 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:17:21,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:17:22,061 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:17:22,061 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:17:22,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:17:22,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:17:22,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c5d6.c.gcp-upstream.internal','N') ;"
2026-09-25 15:17:22,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:17:23,047 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:17:23,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:17:23,296 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:17:23,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:17:23,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:17:23,786 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:17:23,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:17:24.747 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.763 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.780 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.796 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.812 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.830 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.893 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.900 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.914 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.924 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.934 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.949 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.958 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.965 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.972 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:24.979 UTC [20359] WARNING: there is no transaction in progress
2026-09-25 15:17:25,640 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:17:25,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:17:25,895 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:17:25,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:17:26,163 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:17:26,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:17:26,413 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:17:26,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:17:26,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:17:26,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c5d6.c.gcp-upstream.internal';"
2026-09-25 15:17:27,127 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:17:27,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:17:27,383 [I] DB_PATCHES have already been applied
I20260925 15:17:27.388937 8366 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcJTRKe/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:17:27.389001 8366 mini_ranger.cc:193] Using host: 127.8.43.148
I20260925 15:17:27.392095 8366 mini_ranger.cc:227] Ranger admin URL: http://127.8.43.148:55135
Sep 25, 2026 3:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:17:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:17:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:17:28.495051 8366 mini_ranger.cc:160] Time spent starting Ranger: real 8.429s user 0.002s sys 0.006s
Sep 25, 2026 3:17:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:17:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 3:17:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:17:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:17:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:17:49.355371 8366 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.43.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.43.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/rangerkms_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.43.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/rangerkms_127.8.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.43.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.43.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.43.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/HTTP_127.8.43.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.43.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/HTTP_127.8.43.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:17:49 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349469, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:17:49.441316 8366 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:17:49.450129 8366 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:17:50.272115 8366 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:17:50.272204 8366 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms
I20260925 15:17:50.272224 8366 mini_ranger_kms.cc:227] Using postgres at 127.8.43.148:60333
2026-09-25 15:17:50,434 [I] DB FLAVOR :POSTGRES
2026-09-25 15:17:50,434 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:17:50,435 [I] Checking connection
2026-09-25 15:17:50,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:17:50,672 [I] connection success
2026-09-25 15:17:50,672 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:17:50,672 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:17:50,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:17:50,981 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:17:50,981 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:17:50,981 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:17:50,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.43.148:60333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:17:51,235 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:17:51.720757 8366 mini_ranger_kms.cc:310] Created kms service
I20260925 15:17:51.934214 8366 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:17:51.980259 8366 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:17:51 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349469, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:52.366793 8366 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:17:52.366906 8366 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcJTRKe/build/release/bin/postgresql.jar:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcJTRKe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:17:52.366925 8366 mini_ranger_kms.cc:240] Using host: 127.8.43.148
I20260925 15:17:52.370587 8366 mini_ranger_kms.cc:276] Ranger KMS PID: 20539
I20260925 15:17:52.370667 8366 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.43.148:49403
Sep 25, 2026 3:17:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:17:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:17:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:17:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:17:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:17:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:17:54.562856 8366 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.122s user 0.002s sys 0.143s
I20260925 15:17:54.562979 8366 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:17:54.563030 8366 mini_ranger_kms.cc:401] 127.8.43.148:49403/kms/v1/keys
Sep 25 15:17:54 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.43.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:17:55 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.43.148@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
Sep 25 15:17:55 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.43.148@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
Sep 25 15:17:57 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:17:57.863379 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:35711
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:35711
--ranger_config_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:49403/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:57.962910 20590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:57.963106 20590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:57.963127 20590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:57.964273 20590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:57.964316 20590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:57.964332 20590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:57.964345 20590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:57.964358 20590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:57.965880 20590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.43.148:49403/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:35711
--ranger_config_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:57.966104 20590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:57.966337 20590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:57.971349 20595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:57.971333 20598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:57.971408 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:57.971716 20590 server_base.cc:1034] running on GCE node
I20260925 15:17:57.971987 20590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:57.972312 20590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:57.973555 20590 hybrid_clock.cc:648] HybridClock initialized: now 1790349477973509 us; error 36 us; skew 500 ppm
Sep 25 15:17:57 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349477, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:57.982025 20590 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:17:57.982604 20590 webserver.cc:460] Webserver started at http://127.8.43.190:43765/ using document root <none> and password file <none>
I20260925 15:17:57.982869 20590 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:57.982931 20590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:57.983083 20590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:17:57 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349477, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:58.042416 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-c5d6"
server_key: "493cf9ddc7648833ddd3bf834c0e572c"
server_key_iv: "6ed12b431dcf4cf59f105ff086a55fe0"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:58.042901 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-c5d6"
server_key: "493cf9ddc7648833ddd3bf834c0e572c"
server_key_iv: "6ed12b431dcf4cf59f105ff086a55fe0"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:58.044729 20590 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:17:58.060355 20605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:58.060592 20590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:17:58.060678 20590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-c5d6"
server_key: "493cf9ddc7648833ddd3bf834c0e572c"
server_key_iv: "6ed12b431dcf4cf59f105ff086a55fe0"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:58.060758 20590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:58.072962 20590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:58.075060 20590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:58.075203 20590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:58.079936 20590 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:35711
I20260925 15:17:58.079967 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:35711 every 8 connection(s)
I20260925 15:17:58.080355 20590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:58.081117 20658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:58.083585 20658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Bootstrap starting.
I20260925 15:17:58.084178 20658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:58.084520 20658 log.cc:824] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:58.085247 20658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: No bootstrap required, opened a new log
I20260925 15:17:58.087540 20658 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } }
I20260925 15:17:58.087713 20658 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:58.087764 20658 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a90f0ba9b243ec9ef48f1acbc542cd, State: Initialized, Role: FOLLOWER
I20260925 15:17:58.087958 20658 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } }
I20260925 15:17:58.088063 20658 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:58.088111 20658 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:58.088163 20658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:58.088742 20658 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } }
I20260925 15:17:58.088872 20658 leader_election.cc:302] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a90f0ba9b243ec9ef48f1acbc542cd; no voters:
I20260925 15:17:58.089063 20658 leader_election.cc:288] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:58.089112 20661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:58.089291 20661 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 1 LEADER]: Becoming Leader. State: Replica: a6a90f0ba9b243ec9ef48f1acbc542cd, State: Running, Role: LEADER
I20260925 15:17:58.089329 20658 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:58.089442 20661 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } }
I20260925 15:17:58.089802 20662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } } }
I20260925 15:17:58.089807 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 20590
I20260925 15:17:58.089865 20662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:58.089906 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:17:58.090075 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a90f0ba9b243ec9ef48f1acbc542cd. Latest consensus state: current_term: 1 leader_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } } }
I20260925 15:17:58.090133 8366 external_mini_cluster.cc:1428] Setting key 6316d3f7ed4ea219f7f995a966247d06
I20260925 15:17:58.090165 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:58.090550 20665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:58.091040 20665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:58.091974 20658 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:17:58.096824 20665 catalog_manager.cc:1347] Generated new cluster ID: 7fdd293a7b2b44bb8cb40cb3d4031812
I20260925 15:17:58.096988 20665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:17:58 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:58.107872 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:58.090892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.8.43.190:35711)
0925 15:17:58.092469 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:58.092478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:58.092514 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:17:58.094160 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:58.094222 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:58.094376 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:58.094558 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:58.095454 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:58.097494 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:58.098750 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:58.099289 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:58.099449 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:58.102883 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:58.102888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:58.102889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:58.102903 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:58.105573 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:58.106068 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:58.106071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:58.106071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:58.106191 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:58.106447 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:58.106447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:58.106447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:58.106759 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:58.107419 (+ 660us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:58.107493 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:17:58.107636 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1526,"thread_start_us":68,"threads_started":1}
I20260925 15:17:58.110276 20665 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:58.111034 20665 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:58.116516 20665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Generated new TSK 0
I20260925 15:17:58.116721 20665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:17:58.876657 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:49403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35711
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:58.963395 20703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:58.963622 20703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:58.963661 20703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:58.964980 20703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:58.965040 20703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:58.965082 20703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:17:58.966703 20703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.43.148:49403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:58.966979 20703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:58.967272 20703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:58.972468 20709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:58.972491 20708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:58.972590 20703 server_base.cc:1034] running on GCE node
W20260925 15:17:58.972666 20711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:58.973001 20703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:58.973330 20703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:58.974524 20703 hybrid_clock.cc:648] HybridClock initialized: now 1790349478974508 us; error 47 us; skew 500 ppm
Sep 25 15:17:58 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349478, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:58.977567 20703 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:17:58.978000 20703 webserver.cc:460] Webserver started at http://127.8.43.129:43637/ using document root <none> and password file <none>
I20260925 15:17:58.978259 20703 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:58.978327 20703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:58.978480 20703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:17:58 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349478, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:58.994082 20703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "441f5522a0214e268666cf6e10aa04e8"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-c5d6"
server_key: "6cc77766cb1639171c5f86b5ff0abf7e"
server_key_iv: "3bc56c489b7a1b915c7847479ebac5c8"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:58.994524 20703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "441f5522a0214e268666cf6e10aa04e8"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-c5d6"
server_key: "6cc77766cb1639171c5f86b5ff0abf7e"
server_key_iv: "3bc56c489b7a1b915c7847479ebac5c8"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:58.996407 20703 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:17:59.011566 20718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.011859 20703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:17:59.011993 20703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "441f5522a0214e268666cf6e10aa04e8"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-c5d6"
server_key: "6cc77766cb1639171c5f86b5ff0abf7e"
server_key_iv: "3bc56c489b7a1b915c7847479ebac5c8"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.012120 20703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:59.041961 20703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:59.044639 20703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:59.044853 20703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:59.045120 20703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:59.045498 20703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:59.045567 20703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.045614 20703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:59.045646 20703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.052052 20703 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:38903
I20260925 15:17:59.052525 20703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:59.057885 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:38903 every 8 connection(s)
I20260925 15:17:59.062757 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 20703
I20260925 15:17:59.062861 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:17:59.063090 8366 external_mini_cluster.cc:1428] Setting key 46ed5d4ce13c133d3675ac9fd5209554
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349478, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:59.067222 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:59.054488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:54591 (local address 127.8.43.190:35711)
0925 15:17:59.058150 (+ 3662us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:59.058152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:59.058160 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:17:59.058185 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:59.058186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:59.058197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:59.058290 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:59.059001 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.059792 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.060852 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.061142 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.061236 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:59.063622 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:59.063623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:59.063624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:59.063631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:59.065588 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:59.066177 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:59.066178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:59.066179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:59.066251 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:59.066562 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:59.066563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:59.066564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:59.066684 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:59.066759 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:59.067022 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 15:17:59.067121 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3632,"thread_start_us":85,"threads_started":1}
I20260925 15:17:59.067456 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:59.053698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35711 (local address 127.8.43.129:54591)
0925 15:17:59.054521 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:59.054567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:59.055198 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:59.058375 (+ 3177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:59.058379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:59.058574 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:59.058867 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:59.058874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.059916 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.059917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:59.060744 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:59.060746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.061248 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.061248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:59.061344 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:59.061959 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:59.061967 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:59.063533 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:59.065693 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:59.065701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:59.065703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:59.066075 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:59.066329 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:59.066329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:59.066330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:59.066468 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:59.066752 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:59.066758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:59.066892 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 15:17:59.067003 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:17:59.067166 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":560,"threads_started":1}
I20260925 15:17:59.067986 20834 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35711
I20260925 15:17:59.068100 20834 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:59.068284 20834 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:17:59.068838 20622 ts_manager.cc:194] Registered new tserver with Master: 441f5522a0214e268666cf6e10aa04e8 (127.8.43.129:38903)
I20260925 15:17:59.069459 20622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:54591
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:17:59.100023 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:49403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35711
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:59.206931 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:59.207168 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:59.207211 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:59.208372 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:59.208443 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:59.208487 20840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:17:59.209980 20840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.43.148:49403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:59.210237 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:59.210526 20840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:59.215765 20848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:59.215778 20840 server_base.cc:1034] running on GCE node
W20260925 15:17:59.215859 20847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:59.216462 20850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:59.216711 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:59.216980 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:59.218137 20840 hybrid_clock.cc:648] HybridClock initialized: now 1790349479218119 us; error 45 us; skew 500 ppm
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:59.220737 20840 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:17:59.221189 20840 webserver.cc:460] Webserver started at http://127.8.43.130:36297/ using document root <none> and password file <none>
I20260925 15:17:59.221441 20840 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:59.221485 20840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:59.221583 20840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:59.240531 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "750b289e473b45ceaf56f87a5b120f8e"
format_stamp: "Formatted at 2026-09-25 15:17:59 on dist-test-slave-c5d6"
server_key: "8690ccad4c01d8d68c302992d3527d55"
server_key_iv: "a2630d9c3252223a5e204e0e9b71160a"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.240999 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "750b289e473b45ceaf56f87a5b120f8e"
format_stamp: "Formatted at 2026-09-25 15:17:59 on dist-test-slave-c5d6"
server_key: "8690ccad4c01d8d68c302992d3527d55"
server_key_iv: "a2630d9c3252223a5e204e0e9b71160a"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.247403 20840 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.001s
I20260925 15:17:59.279697 20857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.279929 20840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:17:59.280011 20840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "750b289e473b45ceaf56f87a5b120f8e"
format_stamp: "Formatted at 2026-09-25 15:17:59 on dist-test-slave-c5d6"
server_key: "8690ccad4c01d8d68c302992d3527d55"
server_key_iv: "a2630d9c3252223a5e204e0e9b71160a"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.280126 20840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:59.298873 20840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:59.301218 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:59.301397 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:59.301643 20840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:59.302023 20840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:59.302083 20840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.302153 20840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:59.302182 20840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.309504 20840 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:32813
I20260925 15:17:59.309567 20970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:32813 every 8 connection(s)
I20260925 15:17:59.309947 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:59.316859 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 20840
I20260925 15:17:59.316952 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:17:59.317175 8366 external_mini_cluster.cc:1428] Setting key acbae687662bf2fca61a03b8f978577f
I20260925 15:17:59.320089 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:59.310784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:48237 (local address 127.8.43.190:35711)
0925 15:17:59.310835 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:59.310837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:59.311036 (+ 199us) server_negotiation.cc:407] Connection header received
0925 15:17:59.311640 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:59.311642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:59.311658 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:59.311710 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:59.312279 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.312805 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.313792 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.314006 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.314096 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:59.316124 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:59.316125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:59.316126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:59.316134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:59.317975 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:59.318936 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:59.318938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:59.318938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:59.318993 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:59.319312 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:59.319313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:59.319313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:59.319451 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:59.319492 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:59.319849 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 15:17:59.319943 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:17:59.320816 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:59.310743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35711 (local address 127.8.43.130:48237)
0925 15:17:59.310913 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:59.310950 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:59.311531 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:59.311791 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:59.311795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:59.311982 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:59.312178 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:59.312182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.312900 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.312902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:59.313700 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:59.313701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.314116 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.314117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:59.314241 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:59.314744 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:59.314752 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:59.316034 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:59.318532 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:59.318539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:59.318541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:59.318851 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:59.319078 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:59.319078 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:59.319079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:59.319237 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:59.319536 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:59.319542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:59.319704 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 15:17:59.320434 (+ 730us) client_negotiation.cc:240] Negotiation successful
0925 15:17:59.320592 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 15:17:59.321499 20971 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35711
I20260925 15:17:59.321594 20971 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:59.321774 20971 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:17:59.322183 20622 ts_manager.cc:194] Registered new tserver with Master: 750b289e473b45ceaf56f87a5b120f8e (127.8.43.130:32813)
I20260925 15:17:59.322535 20622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:48237
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:17:59.356423 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:49403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35711
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:59.453455 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:59.453691 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:59.453728 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:59.454906 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:59.454977 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:59.455021 20977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:17:59.456727 20977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.43.148:49403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:59.457083 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:59.457381 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:59.463874 20982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:59.464165 20983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:59.464411 20985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:59.465571 20977 server_base.cc:1034] running on GCE node
I20260925 15:17:59.465883 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:59.466147 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:59.467335 20977 hybrid_clock.cc:648] HybridClock initialized: now 1790349479467314 us; error 32 us; skew 500 ppm
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:59.470502 20977 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:17:59.471002 20977 webserver.cc:460] Webserver started at http://127.8.43.131:37035/ using document root <none> and password file <none>
I20260925 15:17:59.471371 20977 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:59.471482 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:59.471673 20977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:59.497174 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "e6caa71c48e2412387f06c900a06e2a2"
format_stamp: "Formatted at 2026-09-25 15:17:59 on dist-test-slave-c5d6"
server_key: "b92029c8863512813c13eccf8b039bf4"
server_key_iv: "96abce79999a7084bfe4a1b614afd25a"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.497637 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "e6caa71c48e2412387f06c900a06e2a2"
format_stamp: "Formatted at 2026-09-25 15:17:59 on dist-test-slave-c5d6"
server_key: "b92029c8863512813c13eccf8b039bf4"
server_key_iv: "96abce79999a7084bfe4a1b614afd25a"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.499545 20977 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:17:59.513250 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.513499 20977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:17:59.513612 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "e6caa71c48e2412387f06c900a06e2a2"
format_stamp: "Formatted at 2026-09-25 15:17:59 on dist-test-slave-c5d6"
server_key: "b92029c8863512813c13eccf8b039bf4"
server_key_iv: "96abce79999a7084bfe4a1b614afd25a"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.513713 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:59.526643 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:59.528748 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:59.528908 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:59.529152 20977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:59.529541 20977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:59.529594 20977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.529660 20977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:59.529690 20977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.546092 20977 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:46847
I20260925 15:17:59.546175 21105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:46847 every 8 connection(s)
I20260925 15:17:59.546657 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:59.552055 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 20977
I20260925 15:17:59.552166 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:17:59.552369 8366 external_mini_cluster.cc:1428] Setting key 930a03e2ac1f38ab1639c6e5a129b1de
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:59.559966 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:59.550926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:57461 (local address 127.8.43.190:35711)
0925 15:17:59.550975 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:59.550978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:59.551246 (+ 268us) server_negotiation.cc:407] Connection header received
0925 15:17:59.551999 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:59.552001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:59.552017 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:59.552061 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:59.552938 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.553704 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.554667 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.554879 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.554965 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:59.557120 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:59.557121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:59.557122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:59.557129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:59.558569 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:59.558979 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:59.558980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:59.558980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:59.559045 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:59.559328 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:59.559329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:59.559329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:59.559443 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:59.559512 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:59.559799 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:17:59.559877 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:17:59.560153 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:59.550933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35711 (local address 127.8.43.131:57461)
0925 15:17:59.551090 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:59.551132 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:59.551832 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:59.552286 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:59.552291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:59.552553 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:59.552837 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:59.552844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.553790 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.553791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:59.554579 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:59.554581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.554986 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.554987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:59.555121 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:59.555723 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:59.555736 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:59.557015 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:59.558669 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:59.558675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:59.558677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:59.558891 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:59.559111 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:59.559112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:59.559113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:59.559267 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:59.559517 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:59.559523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:59.559667 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 15:17:59.559766 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:17:59.559905 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260925 15:17:59.560703 21106 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35711
I20260925 15:17:59.560817 21106 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:59.560997 21106 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:17:59.561378 20622 ts_manager.cc:194] Registered new tserver with Master: e6caa71c48e2412387f06c900a06e2a2 (127.8.43.131:46847)
I20260925 15:17:59.561751 20622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:57461
I20260925 15:17:59.563934 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:17:59.564498 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 20703
I20260925 15:17:59.569959 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 20840
I20260925 15:17:59.576064 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 20977
I20260925 15:17:59.581624 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 20590
I20260925 15:17:59.587488 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:35711
--webserver_interface=127.8.43.190
--webserver_port=43765
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:35711
--ranger_config_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:49403/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.43.148:49403/kms with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:59.669472 21109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:59.669689 21109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:59.669725 21109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:59.670822 21109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:59.670859 21109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:59.670873 21109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:59.670885 21109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:59.670897 21109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:59.672631 21109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.43.148:49403/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:35711
--ranger_config_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=43765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:59.672809 21109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:59.672986 21109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:59.677680 21117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:59.677707 21114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:59.677714 21115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:59.677958 21109 server_base.cc:1034] running on GCE node
I20260925 15:17:59.678159 21109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:59.678386 21109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:59.679524 21109 hybrid_clock.cc:648] HybridClock initialized: now 1790349479679517 us; error 36 us; skew 500 ppm
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:59.681993 21109 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:17:59.682452 21109 webserver.cc:460] Webserver started at http://127.8.43.190:43765/ using document root <none> and password file <none>
I20260925 15:17:59.682685 21109 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:59.682736 21109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:59.739149 21109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:17:59.739914 21125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.740140 21109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:17:59.740238 21109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-c5d6"
server_key: "493cf9ddc7648833ddd3bf834c0e572c"
server_key_iv: "6ed12b431dcf4cf59f105ff086a55fe0"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.740607 21109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:59.748942 21109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:59.750443 21109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:59.750600 21109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:59.754820 21109 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:35711
I20260925 15:17:59.754854 21177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:35711 every 8 connection(s)
I20260925 15:17:59.755257 21109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:59.756417 21178 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:17:59.757023 21178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Bootstrap starting.
I20260925 15:17:59.762884 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 21109
I20260925 15:17:59.763149 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:49403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:38903
--local_ip_for_outbound_sockets=127.8.43.129
--tserver_master_addrs=127.8.43.190:35711
--webserver_port=43637
--webserver_interface=127.8.43.129
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.43.148:49403/kms with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:17:59.765025 21178 log.cc:824] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:59.766676 21178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:17:59.766952 21178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Bootstrap complete.
I20260925 15:17:59.769346 21178 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } }
I20260925 15:17:59.769601 21178 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a90f0ba9b243ec9ef48f1acbc542cd, State: Initialized, Role: FOLLOWER
I20260925 15:17:59.769802 21178 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } }
I20260925 15:17:59.769897 21178 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:59.769971 21178 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:59.770021 21178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:17:59.770993 21178 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } }
I20260925 15:17:59.771111 21178 leader_election.cc:302] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a90f0ba9b243ec9ef48f1acbc542cd; no voters:
I20260925 15:17:59.771330 21178 leader_election.cc:288] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:17:59.771364 21182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:17:59.771473 21182 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [term 2 LEADER]: Becoming Leader. State: Replica: a6a90f0ba9b243ec9ef48f1acbc542cd, State: Running, Role: LEADER
I20260925 15:17:59.771636 21178 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:59.771855 21182 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } }
I20260925 15:17:59.772159 21184 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a90f0ba9b243ec9ef48f1acbc542cd. Latest consensus state: current_term: 2 leader_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } } }
I20260925 15:17:59.772243 21184 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:59.772501 21183 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a90f0ba9b243ec9ef48f1acbc542cd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35711 } } }
I20260925 15:17:59.772693 21183 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:59.772619 21191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:59.773118 21178 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:17:59.773480 21191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:59.776002 21191 catalog_manager.cc:1259] Loaded cluster ID: 7fdd293a7b2b44bb8cb40cb3d4031812
I20260925 15:17:59.776126 21191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:59.777900 21191 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:59.778100 21191 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Loaded TSK: 0
I20260925 15:17:59.778317 21191 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:17:59.877141 21180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:59.877486 21180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:59.877573 21180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:59.885843 21180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:59.885988 21180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:59.886096 21180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:17:59.888507 21180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.43.148:49403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:38903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=43637
--tserver_master_addrs=127.8.43.190:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:17:59.895027 21180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:59.895462 21180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:59.902208 21216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:59.902477 21180 server_base.cc:1034] running on GCE node
W20260925 15:17:59.902655 21215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:59.902931 21218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:59.903278 21180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:59.903550 21180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:59.904707 21180 hybrid_clock.cc:648] HybridClock initialized: now 1790349479904503 us; error 214 us; skew 500 ppm
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:59.908365 21180 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:17:59.908879 21180 webserver.cc:460] Webserver started at http://127.8.43.129:43637/ using document root <none> and password file <none>
I20260925 15:17:59.909161 21180 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:59.909318 21180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:17:59.925387 21180 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:17:59.926556 21226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.926720 21180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:17:59.926795 21180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "441f5522a0214e268666cf6e10aa04e8"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-c5d6"
server_key: "6cc77766cb1639171c5f86b5ff0abf7e"
server_key_iv: "3bc56c489b7a1b915c7847479ebac5c8"
server_key_version: "kuduclusterkey@0"
I20260925 15:17:59.927263 21180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:59.961443 21180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:59.964707 21180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:59.964932 21180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:59.965215 21180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:59.965618 21180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:59.965673 21180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.965709 21180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:59.965767 21180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:59.974015 21180 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:38903
I20260925 15:17:59.974558 21180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:59.975387 21339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:38903 every 8 connection(s)
I20260925 15:17:59.982155 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 21180
I20260925 15:17:59.982458 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:49403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:32813
--local_ip_for_outbound_sockets=127.8.43.130
--tserver_master_addrs=127.8.43.190:35711
--webserver_port=36297
--webserver_interface=127.8.43.130
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.43.148:49403/kms with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:17:59 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:17:59.991312 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:59.975467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35711 (local address 127.8.43.129:36689)
0925 15:17:59.975651 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:59.975696 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:59.976349 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:59.976817 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:59.976822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:59.977064 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:59.977350 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:59.977357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.978621 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.978622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:59.979934 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:59.979938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.980855 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.980857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:59.981021 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:59.981716 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:59.981727 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:59.984530 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:59.987142 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:59.987152 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:59.987155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:59.987512 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:59.989103 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:59.989104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:59.989104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:59.989259 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:59.990501 (+ 1242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:59.990508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:59.990689 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 15:17:59.990803 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:17:59.991002 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":148,"threads_started":1}
I20260925 15:17:59.998127 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:59.975955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:36689 (local address 127.8.43.190:35711)
0925 15:17:59.976153 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:59.976162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:59.976207 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:17:59.976566 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:59.976572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:59.976618 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:59.976737 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:59.977505 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.978507 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.980045 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:59.980717 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:59.980848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:59.984693 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:59.984699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:59.984699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:59.984715 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:59.987026 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:59.988858 (+ 1832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:59.988862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:59.988863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:59.989005 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:59.989948 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:59.989949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:59.989950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:59.990372 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:59.990466 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:59.997701 (+ 7235us) server_negotiation.cc:299] Negotiation successful
0925 15:17:59.997858 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260925 15:17:59.999286 21340 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35711
I20260925 15:17:59.999398 21340 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:59.999606 21340 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
W20260925 15:18:00.000660 21340 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:35711 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:18:00.000751 21340 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:00.000819 21340 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:18:00.001276 21340 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:00.001350 21340 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
W20260925 15:18:00.001801 21340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:18:00.106196 21344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:00.106475 21344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:00.106513 21344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:00.108395 21344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:00.108459 21344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:00.108505 21344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:18:00.110822 21344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.43.148:49403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:32813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=36297
--tserver_master_addrs=127.8.43.190:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:00.111068 21344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:00.111418 21344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:00.117758 21349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:00.117883 21352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:00.118278 21350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:00.118436 21344 server_base.cc:1034] running on GCE node
I20260925 15:18:00.118645 21344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:00.118980 21344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:00.120183 21344 hybrid_clock.cc:648] HybridClock initialized: now 1790349480120152 us; error 46 us; skew 500 ppm
Sep 25 15:18:00 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:00.123507 21344 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:18:00.124071 21344 webserver.cc:460] Webserver started at http://127.8.43.130:36297/ using document root <none> and password file <none>
I20260925 15:18:00.124372 21344 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:00.124466 21344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:18:00 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:18:00.141541 21344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:18:00.142782 21360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:00.143023 21344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:00.143263 21344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "750b289e473b45ceaf56f87a5b120f8e"
format_stamp: "Formatted at 2026-09-25 15:17:59 on dist-test-slave-c5d6"
server_key: "8690ccad4c01d8d68c302992d3527d55"
server_key_iv: "a2630d9c3252223a5e204e0e9b71160a"
server_key_version: "kuduclusterkey@0"
I20260925 15:18:00.143868 21344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:00.172605 21344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:00.174937 21344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:00.175110 21344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:00.175416 21344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:00.175801 21344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:00.175853 21344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:00.175894 21344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:00.175927 21344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:00.183655 21344 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:32813
I20260925 15:18:00.184198 21473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:32813 every 8 connection(s)
I20260925 15:18:00.184237 21344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:18:00 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:00.194407 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 21344
I20260925 15:18:00.194690 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.43.148:49403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:46847
--local_ip_for_outbound_sockets=127.8.43.131
--tserver_master_addrs=127.8.43.190:35711
--webserver_port=37035
--webserver_interface=127.8.43.131
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.43.148:49403/kms with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:00.196580 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:00.185607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:33043 (local address 127.8.43.190:35711)
0925 15:18:00.185656 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:00.185658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:00.185763 (+ 105us) server_negotiation.cc:407] Connection header received
0925 15:18:00.186451 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:00.186452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:00.186464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:00.186532 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:00.187407 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.188217 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.189745 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.190069 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.190146 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:00.192868 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:00.192870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:00.192870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:00.192878 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:00.195110 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:00.195558 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:00.195559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:00.195559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:00.195618 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:00.195921 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:00.195922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:00.195922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:00.196036 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:00.196109 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:00.196409 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 15:18:00.196489 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:18:00.196780 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:00.185322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35711 (local address 127.8.43.130:33043)
0925 15:18:00.185517 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:00.185655 (+ 138us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:00.186326 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:00.186629 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:00.186634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:00.186956 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:00.187274 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:00.187282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.188328 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.188330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:00.189642 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:00.189645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.190192 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.190194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:00.190335 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:00.190951 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:00.190961 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:00.192766 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:00.195237 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:00.195243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:00.195246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:00.195476 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:00.195691 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:00.195692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:00.195692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:00.195836 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:00.196101 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:00.196107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:00.196276 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 15:18:00.196376 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:18:00.196529 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260925 15:18:00.197257 21474 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35711
I20260925 15:18:00.197377 21474 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:00.197602 21474 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
W20260925 15:18:00.198333 21474 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:35711 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:18:00.198419 21474 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:00.198484 21474 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:18:00.198798 21474 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:00.198879 21474 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
W20260925 15:18:00.199157 21474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:18:00.313352 21477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:00.313637 21477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:00.313675 21477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:00.315296 21477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:00.315349 21477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:00.315388 21477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:18:00.317056 21477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.43.148:49403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:46847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=37035
--tserver_master_addrs=127.8.43.190:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:00.317320 21477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:00.317560 21477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:00.322340 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:00.322345 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:00.322356 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:00.323276 21477 server_base.cc:1034] running on GCE node
I20260925 15:18:00.323494 21477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:00.323722 21477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:00.324879 21477 hybrid_clock.cc:648] HybridClock initialized: now 1790349480324867 us; error 32 us; skew 500 ppm
Sep 25 15:18:00 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:00.327450 21477 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:18:00.327931 21477 webserver.cc:460] Webserver started at http://127.8.43.131:37035/ using document root <none> and password file <none>
I20260925 15:18:00.328172 21477 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:00.328226 21477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:18:00 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:18:00.344439 21477 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:00.345521 21494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:00.345723 21477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:18:00.345809 21477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "e6caa71c48e2412387f06c900a06e2a2"
format_stamp: "Formatted at 2026-09-25 15:17:59 on dist-test-slave-c5d6"
server_key: "b92029c8863512813c13eccf8b039bf4"
server_key_iv: "96abce79999a7084bfe4a1b614afd25a"
server_key_version: "kuduclusterkey@0"
I20260925 15:18:00.346252 21477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:00.359840 21477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:00.364316 21477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:00.364480 21477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:00.364703 21477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:00.365015 21477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:00.365067 21477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:00.365108 21477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:00.365135 21477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:00.370916 21477 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:46847
I20260925 15:18:00.371086 21607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:46847 every 8 connection(s)
I20260925 15:18:00.371392 21477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:18:00.372330 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 21477
Sep 25 15:18:00 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:00.380908 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:00.372841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.8.43.190:35711)
0925 15:18:00.372963 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:00.372965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:00.372975 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:00.373028 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:00.373028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:00.373039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:00.373130 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:00.373923 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.374472 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.375698 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.375884 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.375953 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:00.377241 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:00.377242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:00.377242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:00.377250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:00.379010 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:00.379612 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:00.379613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:00.379614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:00.379676 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:00.379881 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:00.379883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:00.379883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:00.380043 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:00.380215 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:00.380693 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 15:18:00.380809 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260925 15:18:00.384027 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:00.372288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:40011 (local address 127.8.43.190:35711)
0925 15:18:00.372351 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:00.372353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:00.372798 (+ 445us) server_negotiation.cc:407] Connection header received
0925 15:18:00.374053 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:00.374054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:00.374066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:00.374102 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:00.374961 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.375638 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.377723 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.378039 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.378105 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:00.380681 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:00.380682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:00.380682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:00.380689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:00.382445 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:00.382922 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:00.382923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:00.382923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:00.382976 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:00.383328 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:00.383329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:00.383329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:00.383460 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:00.383502 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:00.383858 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 15:18:00.383942 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:18:00.385073 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:00.372276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:35711 (local address 127.8.43.131:40011)
0925 15:18:00.372502 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:00.372550 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:00.373823 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:00.374301 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:00.374305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:00.374543 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:00.374848 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:00.374855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.376268 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.376269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:00.377617 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:00.377619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:00.378282 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:00.378283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:00.378438 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:00.378982 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:00.378992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:00.380584 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:00.382536 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:00.382542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:00.382544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:00.382831 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:00.383047 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:00.383047 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:00.383048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:00.383174 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:00.383550 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:00.383556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:00.383702 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 15:18:00.384669 (+ 967us) client_negotiation.cc:240] Negotiation successful
0925 15:18:00.384828 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260925 15:18:00.385741 21608 heartbeater.cc:347] Connected to a master server at 127.8.43.190:35711
I20260925 15:18:00.385878 21608 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:00.386053 21608 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
W20260925 15:18:00.386714 21608 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:35711 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:18:00.386791 21608 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:00.386862 21608 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:18:00.387131 21608 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:00.387202 21608 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
W20260925 15:18:00.387581 21608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:18:00 dist-test-slave-c5d6 krb5kdc[20146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:01.002053 21340 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:01.002247 21340 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:18:01.002660 21142 ts_manager.cc:194] Registered new tserver with Master: 441f5522a0214e268666cf6e10aa04e8 (127.8.43.129:38903)
I20260925 15:18:01.003499 21142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:36689
I20260925 15:18:01.199314 21474 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:01.199487 21474 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:18:01.199934 21142 ts_manager.cc:194] Registered new tserver with Master: 750b289e473b45ceaf56f87a5b120f8e (127.8.43.130:32813)
I20260925 15:18:01.200469 21142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:33043
Sep 25 15:18:01 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:18:01.387784 21608 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:01.387970 21608 heartbeater.cc:511] Master 127.8.43.190:35711 requested a full tablet report, sending...
I20260925 15:18:01.388299 21142 ts_manager.cc:194] Registered new tserver with Master: e6caa71c48e2412387f06c900a06e2a2 (127.8.43.131:46847)
I20260925 15:18:01.388743 21142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:40011
I20260925 15:18:01.396052 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:18:01.404376 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.397031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.8.43.190:35711)
0925 15:18:01.397142 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:01.397145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:01.397197 (+ 52us) server_negotiation.cc:407] Connection header received
0925 15:18:01.397262 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:01.397263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:01.397275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:01.397342 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:01.397936 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.398619 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.399643 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.399933 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.400006 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.401429 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:01.401430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:01.401431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:01.401438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:01.403163 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:01.403554 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:01.403555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:01.403555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:01.403620 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:01.403826 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:01.403827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:01.403827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:01.403959 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:01.404053 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:01.404190 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:18:01.404269 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260925 15:18:01.406833 21142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:01.409241 21142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:01.416925 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.412279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:38903 (local address 127.0.0.1:59290)
0925 15:18:01.412726 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:01.412768 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:01.412840 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:01.413240 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:01.413243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:01.413245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:01.413306 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.413308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.414134 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.414135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.415516 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.415519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.416625 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.416626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.416705 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.416719 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:18:01.416788 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:18:01.416839 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":93,"threads_started":1}
I20260925 15:18:01.417500 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.412551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.8.43.129:38903)
0925 15:18:01.412781 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:01.412783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:01.412982 (+ 199us) server_negotiation.cc:407] Connection header received
0925 15:18:01.413035 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:01.413038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:01.413055 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:01.413115 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:01.413381 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.414048 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.415614 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.416454 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.416523 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.416583 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:01.417378 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 15:18:01.417422 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":67,"threads_started":1}
I20260925 15:18:01.418325 21264 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7f09dc24554d52a220f493007b2cb6 (DEFAULT_TABLE table=test-table [id=86f35bc6574546d9874f0d46d2fd2180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:01.418685 21264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a7f09dc24554d52a220f493007b2cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:01.419399 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.412279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:32813 (local address 127.0.0.1:52520)
0925 15:18:01.415309 (+ 3030us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:01.415325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:01.415422 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:01.415679 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:01.415680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:01.415682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:01.415740 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.415741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.417002 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.417003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.418159 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.418162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.419142 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.419143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.419212 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.419237 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:18:01.419286 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:18:01.419322 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3014,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260925 15:18:01.419627 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.412550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.8.43.130:32813)
0925 15:18:01.412871 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:01.412875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:01.415394 (+ 2519us) server_negotiation.cc:407] Connection header received
0925 15:18:01.415498 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:01.415501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:01.415520 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:01.415596 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:01.415815 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.416910 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.418250 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.418991 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.419057 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.419108 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:01.419499 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 15:18:01.419544 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":66,"threads_started":1}
I20260925 15:18:01.420177 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.412550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.8.43.131:46847)
0925 15:18:01.412720 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:01.412724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:01.415573 (+ 2849us) server_negotiation.cc:407] Connection header received
0925 15:18:01.415679 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:01.415682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:01.415696 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:01.415749 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:01.415980 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.417009 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.418498 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.419199 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.419265 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.419316 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:01.419601 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 15:18:01.419641 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260925 15:18:01.420274 21408 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7f09dc24554d52a220f493007b2cb6 (DEFAULT_TABLE table=test-table [id=86f35bc6574546d9874f0d46d2fd2180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:01.420395 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.412297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:46847 (local address 127.0.0.1:33522)
0925 15:18:01.415511 (+ 3214us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:01.415520 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:01.415598 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:01.415851 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:01.415852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:01.415853 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:01.415902 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.415903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.417092 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.417092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.418408 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.418410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.419369 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.419369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.419517 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.419521 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:18:01.420278 (+ 757us) client_negotiation.cc:240] Negotiation successful
0925 15:18:01.420319 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3199,"thread_start_us":84,"threads_started":1}
I20260925 15:18:01.420611 21408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a7f09dc24554d52a220f493007b2cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:01.422175 21641 tablet_bootstrap.cc:492] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8: Bootstrap starting.
I20260925 15:18:01.422801 21641 tablet_bootstrap.cc:654] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:01.423198 21641 log.cc:824] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:01.423766 21642 tablet_bootstrap.cc:492] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Bootstrap starting.
I20260925 15:18:01.423717 21542 tablet_service.cc:1467] Processing CreateTablet for tablet 0a7f09dc24554d52a220f493007b2cb6 (DEFAULT_TABLE table=test-table [id=86f35bc6574546d9874f0d46d2fd2180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:01.424034 21542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a7f09dc24554d52a220f493007b2cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:01.424333 21642 tablet_bootstrap.cc:654] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:01.424623 21642 log.cc:824] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:01.425397 21641 tablet_bootstrap.cc:492] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8: No bootstrap required, opened a new log
I20260925 15:18:01.425412 21642 tablet_bootstrap.cc:492] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: No bootstrap required, opened a new log
I20260925 15:18:01.425468 21642 ts_tablet_manager.cc:1397] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:18:01.425468 21641 ts_tablet_manager.cc:1397] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:01.427681 21641 raft_consensus.cc:348] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.427803 21645 tablet_bootstrap.cc:492] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: Bootstrap starting.
I20260925 15:18:01.427836 21641 raft_consensus.cc:374] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:01.427870 21641 raft_consensus.cc:729] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441f5522a0214e268666cf6e10aa04e8, State: Initialized, Role: FOLLOWER
I20260925 15:18:01.427997 21641 consensus_queue.cc:260] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.427948 21642 raft_consensus.cc:348] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.428076 21642 raft_consensus.cc:374] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:01.428113 21642 raft_consensus.cc:729] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750b289e473b45ceaf56f87a5b120f8e, State: Initialized, Role: FOLLOWER
I20260925 15:18:01.428233 21641 ts_tablet_manager.cc:1428] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:01.428242 21642 consensus_queue.cc:260] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.428457 21645 tablet_bootstrap.cc:654] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:01.428493 21642 ts_tablet_manager.cc:1428] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:18:01.428625 21474 heartbeater.cc:503] Master 127.8.43.190:35711 was elected leader, sending a full tablet report...
I20260925 15:18:01.428632 21340 heartbeater.cc:503] Master 127.8.43.190:35711 was elected leader, sending a full tablet report...
I20260925 15:18:01.428846 21645 log.cc:824] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:01.431918 21645 tablet_bootstrap.cc:492] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: No bootstrap required, opened a new log
I20260925 15:18:01.432003 21645 ts_tablet_manager.cc:1397] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 15:18:01.434640 21645 raft_consensus.cc:348] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.434849 21645 raft_consensus.cc:374] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:01.434940 21645 raft_consensus.cc:729] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6caa71c48e2412387f06c900a06e2a2, State: Initialized, Role: FOLLOWER
I20260925 15:18:01.435091 21645 consensus_queue.cc:260] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.435374 21608 heartbeater.cc:503] Master 127.8.43.190:35711 was elected leader, sending a full tablet report...
I20260925 15:18:01.435371 21645 ts_tablet_manager.cc:1428] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 15:18:01.435871 21475 tablet.cc:2369] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:18:01.476092 21341 tablet.cc:2369] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:18:01.535845 21647 raft_consensus.cc:482] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:01.536005 21647 raft_consensus.cc:504] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.536437 21647 leader_election.cc:288] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 441f5522a0214e268666cf6e10aa04e8 (127.8.43.129:38903), e6caa71c48e2412387f06c900a06e2a2 (127.8.43.131:46847)
I20260925 15:18:01.541234 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.536930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:41039 (local address 127.8.43.129:38903)
0925 15:18:01.536979 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:01.536981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:01.537367 (+ 386us) server_negotiation.cc:407] Connection header received
0925 15:18:01.537490 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:01.537504 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:01.537517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:01.537551 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:01.537807 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.538621 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.540199 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.540792 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.540858 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.540868 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:01.541115 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 15:18:01.541158 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:18:01.541702 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.536677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:38903 (local address 127.8.43.130:41039)
0925 15:18:01.537287 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:01.537300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:01.537400 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:01.537622 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:01.537623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:01.537625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:01.537725 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.537727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.538741 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.538742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.540081 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.540084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.540957 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.540958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.541027 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.541031 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:18:01.541547 (+ 516us) client_negotiation.cc:240] Negotiation successful
0925 15:18:01.541598 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":114,"threads_started":1}
I20260925 15:18:01.542197 21294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7f09dc24554d52a220f493007b2cb6" candidate_uuid: "750b289e473b45ceaf56f87a5b120f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441f5522a0214e268666cf6e10aa04e8" is_pre_election: true
I20260925 15:18:01.542377 21294 raft_consensus.cc:2454] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 750b289e473b45ceaf56f87a5b120f8e in term 0.
I20260925 15:18:01.542788 21362 leader_election.cc:302] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441f5522a0214e268666cf6e10aa04e8, 750b289e473b45ceaf56f87a5b120f8e; no voters:
I20260925 15:18:01.542943 21647 raft_consensus.cc:2781] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:01.542995 21647 raft_consensus.cc:482] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:01.543017 21647 raft_consensus.cc:3037] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:01.544001 21647 raft_consensus.cc:504] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.544134 21647 leader_election.cc:288] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [CANDIDATE]: Term 1 election: Requested vote from peers 441f5522a0214e268666cf6e10aa04e8 (127.8.43.129:38903), e6caa71c48e2412387f06c900a06e2a2 (127.8.43.131:46847)
I20260925 15:18:01.544409 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.537160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:40647 (local address 127.8.43.131:46847)
0925 15:18:01.537205 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:01.537208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:01.538821 (+ 1613us) server_negotiation.cc:407] Connection header received
0925 15:18:01.538935 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:01.538936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:01.538950 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:01.538982 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:01.539733 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.540388 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.543411 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.543995 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.544046 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.544056 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:01.544295 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 15:18:01.544341 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:18:01.544627 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.537015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:46847 (local address 127.8.43.130:40647)
0925 15:18:01.538741 (+ 1726us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:01.538754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:01.538851 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:01.539578 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:01.539579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:01.539581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:01.539643 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.539645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.541349 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.541350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.543300 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:01.543303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.544138 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.544139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:01.544210 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.544213 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:18:01.544490 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 15:18:01.544530 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1392,"thread_start_us":78,"threads_started":1}
I20260925 15:18:01.544653 21294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7f09dc24554d52a220f493007b2cb6" candidate_uuid: "750b289e473b45ceaf56f87a5b120f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "441f5522a0214e268666cf6e10aa04e8"
I20260925 15:18:01.544734 21294 raft_consensus.cc:3037] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:01.545601 21294 raft_consensus.cc:2454] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 750b289e473b45ceaf56f87a5b120f8e in term 1.
I20260925 15:18:01.545876 21562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7f09dc24554d52a220f493007b2cb6" candidate_uuid: "750b289e473b45ceaf56f87a5b120f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6caa71c48e2412387f06c900a06e2a2" is_pre_election: true
I20260925 15:18:01.546042 21562 raft_consensus.cc:2454] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 750b289e473b45ceaf56f87a5b120f8e in term 0.
I20260925 15:18:01.546154 21561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a7f09dc24554d52a220f493007b2cb6" candidate_uuid: "750b289e473b45ceaf56f87a5b120f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6caa71c48e2412387f06c900a06e2a2"
I20260925 15:18:01.546205 21561 raft_consensus.cc:3037] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:01.546983 21561 raft_consensus.cc:2454] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 750b289e473b45ceaf56f87a5b120f8e in term 1.
I20260925 15:18:01.547245 21364 leader_election.cc:302] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 750b289e473b45ceaf56f87a5b120f8e, e6caa71c48e2412387f06c900a06e2a2; no voters:
I20260925 15:18:01.547616 21647 raft_consensus.cc:2781] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:01.547768 21647 raft_consensus.cc:686] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 1 LEADER]: Becoming Leader. State: Replica: 750b289e473b45ceaf56f87a5b120f8e, State: Running, Role: LEADER
I20260925 15:18:01.547851 21647 consensus_queue.cc:237] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } }
I20260925 15:18:01.548645 21142 catalog_manager.cc:5495] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 750b289e473b45ceaf56f87a5b120f8e (127.8.43.130). New cstate: current_term: 1 leader_uuid: "750b289e473b45ceaf56f87a5b120f8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "750b289e473b45ceaf56f87a5b120f8e" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32813 } health_report { overall_health: HEALTHY } } }
I20260925 15:18:01.565353 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:01.561999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.8.43.130:32813)
0925 15:18:01.562067 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:01.562069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:01.562079 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:01.562119 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:01.562120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:01.562146 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:01.562239 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:01.562414 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.563190 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.564325 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:01.564640 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:01.564734 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:01.564928 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:01.565076 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:01.565178 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:01.565243 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 15:18:01.565268 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:18:01.567582 21561 raft_consensus.cc:1261] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 1 FOLLOWER]: Refusing update from remote peer 750b289e473b45ceaf56f87a5b120f8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:01.567780 21294 raft_consensus.cc:1261] T 0a7f09dc24554d52a220f493007b2cb6 P 441f5522a0214e268666cf6e10aa04e8 [term 1 FOLLOWER]: Refusing update from remote peer 750b289e473b45ceaf56f87a5b120f8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:01.568162 21652 consensus_queue.cc:1035] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6caa71c48e2412387f06c900a06e2a2" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:01.568301 21647 consensus_queue.cc:1035] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "441f5522a0214e268666cf6e10aa04e8" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 25 15:18:01 dist-test-slave-c5d6 krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for HTTP/127.8.43.148@KRBTEST.COM
I20260925 15:18:01.582392 21658 mvcc.cc:205] Tried to move back new op lower bound from 7333271476497776640 to 7333271476420427776. Current Snapshot: MvccSnapshot[applied={T|T < 7333271476497776640}]
I20260925 15:18:01.585167 21141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:01.585270 21141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:01.589987 21141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a90f0ba9b243ec9ef48f1acbc542cd: Sending DeleteTablet for 3 replicas of tablet 0a7f09dc24554d52a220f493007b2cb6
I20260925 15:18:01.590365 21542 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7f09dc24554d52a220f493007b2cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:01 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:33522
I20260925 15:18:01.590378 21408 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7f09dc24554d52a220f493007b2cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:01 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:52520
I20260925 15:18:01.590723 21264 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a7f09dc24554d52a220f493007b2cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:01 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:59290
I20260925 15:18:01.591058 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 21180
I20260925 15:18:01.607933 21665 tablet_replica.cc:330] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: stopping tablet replica
I20260925 15:18:01.608177 21665 raft_consensus.cc:2229] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:18:01.608428 21665 raft_consensus.cc:2258] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:01.609084 21665 ts_tablet_manager.cc:1905] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:01.615139 21666 tablet_replica.cc:330] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: stopping tablet replica
W20260925 15:18:01.615164 21127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:38903 (error 108)
I20260925 15:18:01.615309 21666 raft_consensus.cc:2229] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:01.615447 21666 raft_consensus.cc:2258] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:01.617854 21666 ts_tablet_manager.cc:1905] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:01.619616 21666 ts_tablet_manager.cc:1918] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:01.619802 21665 ts_tablet_manager.cc:1918] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:01.619926 21665 log.cc:1196] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/0a7f09dc24554d52a220f493007b2cb6
I20260925 15:18:01.620321 21665 ts_tablet_manager.cc:1939] T 0a7f09dc24554d52a220f493007b2cb6 P 750b289e473b45ceaf56f87a5b120f8e: Deleting consensus metadata
I20260925 15:18:01.620729 21666 log.cc:1196] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/0a7f09dc24554d52a220f493007b2cb6
I20260925 15:18:01.620820 21128 catalog_manager.cc:4841] TS 750b289e473b45ceaf56f87a5b120f8e (127.8.43.130:32813): tablet 0a7f09dc24554d52a220f493007b2cb6 (table test-table [id=86f35bc6574546d9874f0d46d2fd2180]) successfully deleted
I20260925 15:18:01.621030 21666 ts_tablet_manager.cc:1939] T 0a7f09dc24554d52a220f493007b2cb6 P e6caa71c48e2412387f06c900a06e2a2: Deleting consensus metadata
I20260925 15:18:01.621480 21129 catalog_manager.cc:4841] TS e6caa71c48e2412387f06c900a06e2a2 (127.8.43.131:46847): tablet 0a7f09dc24554d52a220f493007b2cb6 (table test-table [id=86f35bc6574546d9874f0d46d2fd2180]) successfully deleted
W20260925 15:18:01.623472 21636 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:01.615667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:38903 (local address 127.0.0.1:59302)
0925 15:18:01.615731 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:01.615745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:01.615792 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:01.623418 (+ 7626us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:38903: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:18:01.623613 21127 catalog_manager.cc:4584] TS 441f5522a0214e268666cf6e10aa04e8 (127.8.43.129:38903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a7f09dc24554d52a220f493007b2cb6: Network error: Client connection negotiation failed: client connection to 127.8.43.129:38903: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:18:01.627411 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 21344
I20260925 15:18:01.635274 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 21477
I20260925 15:18:01.641388 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 21109
I20260925 15:18:01.650897 8366 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:18:01.994992 8366 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:18:01.995095 8366 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:18:02.339673 8366 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:18:02.339 UTC [20176] LOG: received smart shutdown request
2026-09-25 15:18:02.343 UTC [20176] LOG: background worker "logical replication launcher" (PID 20185) exited with exit code 1
2026-09-25 15:18:02.344 UTC [20180] LOG: shutting down
2026-09-25 15:18:02.493 UTC [20176] LOG: database system is shut down
2026-09-25T15:18:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47314 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:02 dist-test-slave-c5d6 krb5kdc[21678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:02 dist-test-slave-c5d6 krb5kdc[21678](info): set up 2 sockets
Sep 25 15:18:02 dist-test-slave-c5d6 krb5kdc[21678](info): commencing operation
krb5kdc: starting...
W20260925 15:18:03.647147 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:18:03 dist-test-slave-c5d6 krb5kdc[21678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:18:03.765686 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:46255
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:46255,127.8.43.189:42251,127.8.43.188:41049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:03.849273 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:03.849534 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:03.849572 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:03.850705 21691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:03.850777 21691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:03.850804 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:03.850833 21691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:03.850862 21691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:03.852336 21691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:46255,127.8.43.189:42251,127.8.43.188:41049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:46255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:03.852624 21691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:03.852865 21691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:03.857817 21696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:03.857988 21691 server_base.cc:1034] running on GCE node
W20260925 15:18:03.857875 21699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:03.857853 21697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:03.858379 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:03.858719 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:03.859923 21691 hybrid_clock.cc:648] HybridClock initialized: now 1790349483859898 us; error 38 us; skew 500 ppm
Sep 25 15:18:03 dist-test-slave-c5d6 krb5kdc[21678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:03.866207 21691 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:18:03.866645 21691 webserver.cc:460] Webserver started at http://127.8.43.190:42425/ using document root <none> and password file <none>
I20260925 15:18:03.866883 21691 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:03.866938 21691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:03.867085 21691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:03.868386 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "cf2a7aadb6d449a580be120fbdfb3bcd"
format_stamp: "Formatted at 2026-09-25 15:18:03 on dist-test-slave-c5d6"
server_key: "77f2cb7935024bce0e93af2d34e39866"
server_key_iv: "e48c8bf4bd267ea8caff78dba669f083"
server_key_version: "encryptionkey@0"
I20260925 15:18:03.868752 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "cf2a7aadb6d449a580be120fbdfb3bcd"
format_stamp: "Formatted at 2026-09-25 15:18:03 on dist-test-slave-c5d6"
server_key: "77f2cb7935024bce0e93af2d34e39866"
server_key_iv: "e48c8bf4bd267ea8caff78dba669f083"
server_key_version: "encryptionkey@0"
I20260925 15:18:03.870543 21691 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:18:03.871368 21706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:03.871572 21691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:03.871690 21691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "cf2a7aadb6d449a580be120fbdfb3bcd"
format_stamp: "Formatted at 2026-09-25 15:18:03 on dist-test-slave-c5d6"
server_key: "77f2cb7935024bce0e93af2d34e39866"
server_key_iv: "e48c8bf4bd267ea8caff78dba669f083"
server_key_version: "encryptionkey@0"
I20260925 15:18:03.871765 21691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:03.896332 21691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:03.898324 21691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:03.898489 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:03.902506 21691 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:46255
I20260925 15:18:03.902563 21758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:46255 every 8 connection(s)
I20260925 15:18:03.902920 21691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:03.903864 21759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:03.905434 21759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:03.911026 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 21691
I20260925 15:18:03.911126 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
Sep 25 15:18:03 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:03.911391 8366 external_mini_cluster.cc:1428] Setting key 5dd8e1531f2861e424b985071ec9b24c
I20260925 15:18:03.915594 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:03.906136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.8.43.190:46255)
0925 15:18:03.906325 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:03.906332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:03.906499 (+ 167us) server_negotiation.cc:407] Connection header received
0925 15:18:03.906985 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:03.906987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:03.907003 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:03.907084 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:03.907663 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:03.908484 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:03.909352 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:03.909659 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:03.909737 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:03.911894 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:03.911902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:03.911903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:03.911912 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:03.913706 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:03.914246 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:03.914249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:03.914250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:03.914360 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:03.914618 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:03.914619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:03.914619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:03.914935 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:03.915039 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:03.915241 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:18:03.915469 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260925 15:18:03.915599 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:03.906113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46255 (local address 127.0.0.1:35456)
0925 15:18:03.906392 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:03.906418 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:03.906855 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:03.907110 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:03.907112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:03.907410 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:03.907569 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:03.907574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:03.908575 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:03.908576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:03.909276 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:03.909278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:03.909737 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:03.909738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:03.909796 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:03.910289 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:03.910297 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:03.911772 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:03.913800 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:03.913807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:03.913809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:03.914059 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:03.914457 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:03.914458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:03.914458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:03.914543 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:03.915019 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:03.915025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:03.915080 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:18:03.915156 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:18:03.915363 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":122,"threads_started":1}
I20260925 15:18:03.916378 21759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:03.916772 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:03.916621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39248)
0925 15:18:03.916678 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:03.916734 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 15:18:03.916939 21708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
I20260925 15:18:03.917407 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:03.917282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39260)
0925 15:18:03.917338 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:03.917373 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:18:03.917526 21759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:42251: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.43.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab.
Entry for principal kudu/127.8.43.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab.
I20260925 15:18:03.937806 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.189:42251
--webserver_interface=127.8.43.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab
--principal=kudu/127.8.43.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:46255,127.8.43.189:42251,127.8.43.188:41049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:03.966786 21759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } attempt: 1
I20260925 15:18:03.967339 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:03.967175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39264)
0925 15:18:03.967257 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:03.967305 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:18:03.967739 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:03.967637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39276)
0925 15:18:03.967684 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:03.967715 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:18:03.967839 21759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:42251: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
W20260925 15:18:04.018944 21766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:04.019150 21766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:04.019171 21766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:04.020326 21766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:04.020371 21766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:04.020386 21766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:04.020398 21766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:04.020411 21766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:04.021889 21766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:46255,127.8.43.189:42251,127.8.43.188:41049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab
--principal=kudu/127.8.43.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.189:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.43.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:04.022080 21766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:04.022246 21766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:04.027073 21772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:04.027275 21766 server_base.cc:1034] running on GCE node
W20260925 15:18:04.027136 21771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:04.027073 21774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:04.027773 21766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:04.028054 21766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:04.029211 21766 hybrid_clock.cc:648] HybridClock initialized: now 1790349484029199 us; error 33 us; skew 500 ppm
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:04.031868 21766 init.cc:377] Logged in from keytab as kudu/127.8.43.189@KRBTEST.COM (short username kudu)
I20260925 15:18:04.032255 21766 webserver.cc:460] Webserver started at http://127.8.43.189:36225/ using document root <none> and password file <none>
I20260925 15:18:04.032503 21766 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:04.032573 21766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:04.032727 21766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:04.033808 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/data/instance:
uuid: "ab9d2aeed1b743fdbc868ea0877b876e"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-c5d6"
server_key: "ef036eb9b7b8c83487f98d0237f6a44d"
server_key_iv: "fbb56ec512c8f76935421221496159ea"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.034175 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/wal/instance:
uuid: "ab9d2aeed1b743fdbc868ea0877b876e"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-c5d6"
server_key: "ef036eb9b7b8c83487f98d0237f6a44d"
server_key_iv: "fbb56ec512c8f76935421221496159ea"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.035640 21766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:18:04.035979 21759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } attempt: 2
I20260925 15:18:04.036417 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.036291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39278)
0925 15:18:04.036350 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.036386 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:18:04.036600 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:04.036827 21766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:18:04.036844 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.036707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39282)
0925 15:18:04.036775 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.036810 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:18:04.036937 21759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:42251: Network error: Client connection negotiation failed: client connection to 127.8.43.189:42251: connect: Connection refused (error 111)
I20260925 15:18:04.036926 21766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/wal
uuid: "ab9d2aeed1b743fdbc868ea0877b876e"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-c5d6"
server_key: "ef036eb9b7b8c83487f98d0237f6a44d"
server_key_iv: "fbb56ec512c8f76935421221496159ea"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.037009 21766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:04.058216 21766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:04.060564 21766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:04.060674 21766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:04.064620 21766 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.189:42251
I20260925 15:18:04.064670 21833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.189:42251 every 8 connection(s)
I20260925 15:18:04.065038 21766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
I20260925 15:18:04.065853 21834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:04.067384 21834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:04.072475 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 21766
I20260925 15:18:04.072593 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-1/wal/instance
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:04.072811 8366 external_mini_cluster.cc:1428] Setting key c52944939d92e21eadd3a7281ddc8e67
I20260925 15:18:04.076331 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.068058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.8.43.190:46255)
0925 15:18:04.068113 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.068115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.068329 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:18:04.068840 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.068841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.068850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.068909 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.069481 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.069972 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.070903 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.071106 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.071157 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.073232 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.073233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.073234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.073241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.074739 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.075212 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.075213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.075214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.075296 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.075564 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.075565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.075566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.075677 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.075759 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.076073 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.076192 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:18:04.076437 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.068032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46255 (local address 127.0.0.1:35470)
0925 15:18:04.068204 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.068243 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:04.068756 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:04.068991 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:04.068994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:04.069224 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:04.069407 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.069411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.070055 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.070056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.070831 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.070833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.071176 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.071176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.071339 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.071733 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:04.071741 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:04.073136 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:04.074829 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.074836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.074838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.075130 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.075363 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.075364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.075364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.075479 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.075758 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:04.075765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:04.075899 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 15:18:04.076025 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:18:04.076200 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260925 15:18:04.077103 21834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.43.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.188@KRBTEST.COM" created.
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:18:04.086364 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.077364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39298)
0925 15:18:04.077415 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.077426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:04.077466 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:04.077732 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:04.077733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:04.077866 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:04.077909 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.077910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.078795 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.078796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.079356 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.079357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.079830 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.079831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.079894 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.080291 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:04.080296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:04.083438 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:04.085408 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.085409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.085410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.085565 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.085781 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.085781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.085782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.085829 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.086146 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:04.086147 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:04.086181 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:18:04.086233 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:18:04.086304 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:18:04.086541 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.077423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.8.43.189:42251)
0925 15:18:04.077597 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.077601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.077616 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:18:04.077632 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.077633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.077642 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.077718 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.078000 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.078696 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.079443 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.079741 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.079830 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.083517 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.083524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.083525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.083546 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.085294 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.085646 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.085649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.085649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.085715 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.085896 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.085897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.085897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.086080 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.086170 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.086268 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.086387 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260925 15:18:04.087399 21834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:04.087715 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.087607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35236)
0925 15:18:04.087650 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.087699 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:18:04.087875 21783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
I20260925 15:18:04.088282 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.088196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35242)
0925 15:18:04.088244 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.088267 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:18:04.088411 21834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:41049: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab.
Entry for principal kudu/127.8.43.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab.
I20260925 15:18:04.097258 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.188:41049
--webserver_interface=127.8.43.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab
--principal=kudu/127.8.43.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:46255,127.8.43.189:42251,127.8.43.188:41049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:04.128070 21759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } attempt: 3
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:18:04.135900 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.128423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39312)
0925 15:18:04.128490 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.128507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:04.128573 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:04.128727 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:04.128728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:04.128898 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:04.128976 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.128978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.129715 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.129716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.130252 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.130254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.130778 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.130780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.130843 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.131507 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:04.131513 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:04.133140 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:04.134893 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.134894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.134894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.135078 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.135305 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.135306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.135306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.135370 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.135636 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:04.135636 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:04.135677 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:18:04.135734 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:18:04.135817 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:18:04.135939 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.128509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.8.43.189:42251)
0925 15:18:04.128578 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.128581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.128610 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:18:04.128640 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.128641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.128660 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.128719 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.129072 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.129626 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.130350 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.130685 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.130775 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.133238 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.133240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.133240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.133248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.134800 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.135154 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.135155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.135155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.135216 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.135439 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.135440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.135440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.135569 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.135634 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.135774 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.135852 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:18:04.136452 21759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:04.136816 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.136707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35258)
0925 15:18:04.136759 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.136794 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:18:04.137133 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.137071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35274)
0925 15:18:04.137096 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.137118 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 15:18:04.137245 21759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:41049: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
I20260925 15:18:04.137646 21834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } attempt: 1
I20260925 15:18:04.138000 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.137896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35278)
0925 15:18:04.137948 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.137977 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:18:04.138327 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.138253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35294)
0925 15:18:04.138288 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.138311 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 15:18:04.138427 21834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:41049: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
I20260925 15:18:04.168406 21759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } attempt: 1
I20260925 15:18:04.168902 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.168743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35300)
0925 15:18:04.168817 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.168863 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:18:04.169291 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.169185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35302)
0925 15:18:04.169234 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.169264 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 15:18:04.169395 21759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:41049: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
W20260925 15:18:04.179198 21841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:04.179430 21841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:04.179471 21841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:04.180621 21841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:04.180660 21841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:04.180675 21841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:04.180719 21841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:04.180747 21841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:04.182271 21841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:46255,127.8.43.189:42251,127.8.43.188:41049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab
--principal=kudu/127.8.43.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.188:41049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.43.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:04.182510 21841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:04.182734 21841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:04.187424 21847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:04.187397 21849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:04.187454 21841 server_base.cc:1034] running on GCE node
W20260925 15:18:04.187465 21846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:04.187953 21841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:04.188225 21841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:04.189373 21841 hybrid_clock.cc:648] HybridClock initialized: now 1790349484189357 us; error 29 us; skew 500 ppm
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:04.191836 21841 init.cc:377] Logged in from keytab as kudu/127.8.43.188@KRBTEST.COM (short username kudu)
I20260925 15:18:04.192279 21841 webserver.cc:460] Webserver started at http://127.8.43.188:44311/ using document root <none> and password file <none>
I20260925 15:18:04.192476 21841 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:04.192515 21841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:04.192602 21841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:04.193572 21841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/data/instance:
uuid: "73755e2ffa1947bcb8bfe37f2e6d7809"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-c5d6"
server_key: "077eb55f41dd848f98e9204f9f8cacdc"
server_key_iv: "ea71505666aacabb1079c500577a1103"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.193866 21841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/wal/instance:
uuid: "73755e2ffa1947bcb8bfe37f2e6d7809"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-c5d6"
server_key: "077eb55f41dd848f98e9204f9f8cacdc"
server_key_iv: "ea71505666aacabb1079c500577a1103"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.195091 21841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:18:04.196013 21856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:04.196208 21841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:18:04.196297 21841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/wal
uuid: "73755e2ffa1947bcb8bfe37f2e6d7809"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-c5d6"
server_key: "077eb55f41dd848f98e9204f9f8cacdc"
server_key_iv: "ea71505666aacabb1079c500577a1103"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.196408 21841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:04.206578 21834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } attempt: 2
I20260925 15:18:04.206981 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.206841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35314)
0925 15:18:04.206904 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.206945 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:18:04.207340 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.207271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35322)
0925 15:18:04.207299 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.207323 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 15:18:04.207443 21834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:41049: Network error: Client connection negotiation failed: client connection to 127.8.43.188:41049: connect: Connection refused (error 111)
I20260925 15:18:04.208029 21841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:04.210269 21841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:04.210422 21841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:04.214394 21841 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.188:41049
I20260925 15:18:04.214449 21908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.188:41049 every 8 connection(s)
I20260925 15:18:04.214814 21841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
I20260925 15:18:04.215777 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:04.217276 21909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:04.221549 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 21841
I20260925 15:18:04.221661 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349306597827-8366-0/minicluster-data/master-2/wal/instance
I20260925 15:18:04.221854 8366 external_mini_cluster.cc:1428] Setting key 2d549f756bf7aea5b2c30a65b5a686f6
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:04.227274 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.217957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.8.43.190:46255)
0925 15:18:04.218022 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.218025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.218254 (+ 229us) server_negotiation.cc:407] Connection header received
0925 15:18:04.218669 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.218670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.218679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.218735 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.219351 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.220401 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.221349 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.221660 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.221732 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.224364 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.224365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.224365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.224372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.225799 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.226248 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.226250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.226250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.226298 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.226574 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.226588 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.226588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.226728 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.226819 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.227061 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.227153 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:18:04.227326 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.217951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46255 (local address 127.0.0.1:35480)
0925 15:18:04.218121 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.218180 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:04.218594 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:04.218782 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:04.218785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:04.219009 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:04.219199 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.219203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.220496 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.220497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.221271 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.221273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.221763 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.221764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.221875 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.222377 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:04.222387 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:04.224267 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:04.225867 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.225871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.225873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.226159 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.226386 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.226387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.226387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.226509 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.226794 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:04.226817 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:04.226932 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 15:18:04.227010 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:18:04.227113 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260925 15:18:04.227923 21909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:18:04.231333 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.222258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.8.43.188:41049)
0925 15:18:04.222423 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.222434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.222546 (+ 112us) server_negotiation.cc:407] Connection header received
0925 15:18:04.222585 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.222587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.222606 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.222660 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.223426 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.224482 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.225602 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.225935 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.226007 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.228570 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.228576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.228576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.228581 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.230051 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.230491 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.230494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.230494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.230549 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.230776 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.230777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.230778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.230922 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.231092 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.231136 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.231209 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:18:04.234336 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.228194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:42251 (local address 127.0.0.1:39314)
0925 15:18:04.228304 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.228315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:04.228369 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:04.228497 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:04.228498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:04.228635 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:04.228679 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.228680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.229296 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.229297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.229766 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.229767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.230142 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.230143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.230217 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.230597 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:04.230602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:04.231842 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:04.233441 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.233442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.233442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.233596 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.233786 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.233786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.233786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.233832 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.234067 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:04.234067 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:04.234095 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:18:04.234168 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:18:04.234248 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 15:18:04.234386 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.228252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.8.43.189:42251)
0925 15:18:04.228297 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.228299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.228354 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:18:04.228432 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.228433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.228441 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.228491 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.228758 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.229219 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.229858 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.230062 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.230141 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.231902 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.231903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.231904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.231911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.233363 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.233663 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.233664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.233664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.233723 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.233889 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.233890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.233890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.234010 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.234077 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.234213 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.234288 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:18:04.234908 21909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:18:04.239837 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.232102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.8.43.190:46255)
0925 15:18:04.232138 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.232140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.232151 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:18:04.232175 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.232176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.232184 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.232237 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.232782 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.233260 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.234147 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.234342 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.234413 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.237022 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.237023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.237024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.237031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.238597 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.238943 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.238944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.238945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.239012 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.239245 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.239246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.239246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.239369 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.239574 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.239613 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.239696 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:18:04.241905 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.235151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35344)
0925 15:18:04.235202 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.235213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:04.235297 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:04.235404 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:04.235404 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:04.235556 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:04.235592 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.235594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.236184 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.236185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.236694 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.236696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.237043 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.237044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.237087 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.237402 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:04.237407 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:04.238537 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:04.240315 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.240316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.240317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.240563 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.241237 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.241238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.241238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.241292 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.241559 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:04.241560 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:04.241598 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:18:04.241691 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:18:04.241786 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:18:04.241951 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.235234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.8.43.188:41049)
0925 15:18:04.235274 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.235276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.235284 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:18:04.235348 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.235348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.235356 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.235399 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.235664 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.236097 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.236766 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.236966 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.237046 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.238622 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.238624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.238624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.238630 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.240238 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.240653 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.240654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.240655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.240712 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.241372 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.241373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.241374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.241486 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.241567 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.241713 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.241840 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":24}
I20260925 15:18:04.243824 21909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809: Bootstrap starting.
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:18:04.244534 21759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } attempt: 2
I20260925 15:18:04.244859 21909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:04.245218 21909 log.cc:824] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:04.246129 21909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809: No bootstrap required, opened a new log
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:18:04.248556 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.240595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.8.43.189:42251)
0925 15:18:04.240633 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.240635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.240642 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:18:04.240704 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.240704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.240711 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.240754 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.241218 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.241639 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.242293 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.242625 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.242683 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.245111 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.245112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.245113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.245121 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.246963 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.247404 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.247405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.247405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.247470 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.247673 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.247674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.247675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.247800 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.248309 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.248378 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.248467 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:18:04.248515 21909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.248908 21909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:04.249116 21909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73755e2ffa1947bcb8bfe37f2e6d7809, State: Initialized, Role: FOLLOWER
I20260925 15:18:04.249270 21909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.249608 21909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:04.249861 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.249970 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:04.250675 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 15:18:04.251439 21926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:18:04.251487 21926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:18:04.252058 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.244752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35352)
0925 15:18:04.244796 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.244806 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:04.244844 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:04.245053 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:04.245054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:04.245220 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:04.245261 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.245263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.246137 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.246139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.246678 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.246680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.247128 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.247129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.247189 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.247692 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:04.247696 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:04.248944 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:04.250833 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.250833 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.250834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.251061 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.251354 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.251355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.251355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.251418 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.251739 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:04.251740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:04.251789 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:18:04.251867 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:18:04.251959 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 15:18:04.252367 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.244808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.8.43.188:41049)
0925 15:18:04.244847 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.244849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.244889 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:18:04.244911 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.244912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.244921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.245018 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.245353 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.246050 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.246793 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.247054 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.247120 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.249016 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.249017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.249017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.249024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.250740 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.251149 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.251151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.251151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.251271 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.251493 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.251494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.251495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.251656 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.251732 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.252214 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.252287 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:18:04.254439 21759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd: Bootstrap starting.
I20260925 15:18:04.255307 21841 master_runner.cc:186] Error getting master registration for 127.8.43.190:46255: OK, instance_id { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" instance_seqno: 1790349483901418 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:18:04.257891 21841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:18:04.258067 21759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:04.258440 21759 log.cc:824] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:04.260020 21759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd: No bootstrap required, opened a new log
I20260925 15:18:04.261061 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.253016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.8.43.190:46255)
0925 15:18:04.253087 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.253089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.253377 (+ 288us) server_negotiation.cc:407] Connection header received
0925 15:18:04.253497 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.253498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.253508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.253536 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.254323 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.255148 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.256299 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.256587 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.256627 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.258121 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.258122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.258123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.258129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.259830 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.260211 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.260212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.260213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.260268 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.260488 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.260489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.260489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.260607 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.260676 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.260898 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.260974 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:18:04.262204 21759 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.262333 21759 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:04.262369 21759 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2a7aadb6d449a580be120fbdfb3bcd, State: Initialized, Role: FOLLOWER
I20260925 15:18:04.262482 21759 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.262779 21936 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.262863 21936 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:04.263511 21759 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:04.265848 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.253419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.8.43.189:42251)
0925 15:18:04.253457 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.253459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.253979 (+ 520us) server_negotiation.cc:407] Connection header received
0925 15:18:04.254081 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.254082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.254093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.254127 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.254956 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.255655 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.256629 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.257174 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.257222 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.262888 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.262889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.262890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.262897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.264601 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.264979 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.264980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.264981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.265041 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.265249 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.265250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.265251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.265417 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.265554 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.265672 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.265757 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:18:04.267668 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.253720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.8.43.188:41049)
0925 15:18:04.253755 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.253756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.258090 (+ 4334us) server_negotiation.cc:407] Connection header received
0925 15:18:04.259393 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.259394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.259405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.259459 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.260113 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.260755 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.262197 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.262470 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.262518 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.264197 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.264198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.264198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.264205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.266341 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.266769 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.266770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.266771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.266829 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.267047 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.267047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.267048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.267160 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.267282 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.267507 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.267582 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:18:04.268107 21691 master_runner.cc:186] Error getting master registration for 127.8.43.189:42251: OK, instance_id { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" instance_seqno: 1790349484063429 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:18:04.270655 21691 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:18:04.272113 21947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:18:04.272167 21947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:18:04.298605 21834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } attempt: 3
Sep 25 15:18:04 dist-test-slave-c5d6 krb5kdc[21678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:18:04.307144 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.298982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:41049 (local address 127.0.0.1:35364)
0925 15:18:04.299038 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:04.299051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:04.299092 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:04.299312 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:04.299313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:04.299527 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:04.299604 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.299606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.300528 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.300529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.301305 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:04.301307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.301874 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.301876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:04.301940 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.302468 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:04.302474 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:04.304073 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:04.306013 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.306015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.306015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.306230 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.306454 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:04.306455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:04.306456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:04.306533 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:04.306824 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:04.306825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:04.306872 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:18:04.306944 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:18:04.307039 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:18:04.307168 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.299058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.8.43.188:41049)
0925 15:18:04.299099 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.299101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.299115 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:18:04.299144 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.299145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.299154 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.299256 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.299694 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.300427 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.301403 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.301718 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.301795 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.304169 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.304171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.304171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:04.304179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.305910 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.306314 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.306315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.306315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.306371 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:04.306613 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:04.306614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:04.306614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:04.306749 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.306824 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.306988 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.307065 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:18:04.309109 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e: Bootstrap starting.
I20260925 15:18:04.309906 21834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:04.310228 21834 log.cc:824] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:04.311664 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e: No bootstrap required, opened a new log
I20260925 15:18:04.313733 21834 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.313887 21834 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:04.313925 21834 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9d2aeed1b743fdbc868ea0877b876e, State: Initialized, Role: FOLLOWER
I20260925 15:18:04.314088 21834 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.314410 21834 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:04.314378 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.314594 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:18:04.316321 21961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:18:04.316421 21961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:18:04.318850 21766 master_runner.cc:263] No leader master found from master ab9d2aeed1b743fdbc868ea0877b876e
I20260925 15:18:04.318900 21766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:18:04.320994 21915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:04.321076 21915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.321280 21915 leader_election.cc:288] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf2a7aadb6d449a580be120fbdfb3bcd (127.8.43.190:46255), ab9d2aeed1b743fdbc868ea0877b876e (127.8.43.189:42251)
I20260925 15:18:04.321805 21808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" is_pre_election: true
I20260925 15:18:04.321962 21808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73755e2ffa1947bcb8bfe37f2e6d7809 in term 0.
I20260925 15:18:04.322252 21858 leader_election.cc:302] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73755e2ffa1947bcb8bfe37f2e6d7809, ab9d2aeed1b743fdbc868ea0877b876e; no voters:
I20260925 15:18:04.322510 21733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" is_pre_election: true
I20260925 15:18:04.322647 21733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73755e2ffa1947bcb8bfe37f2e6d7809 in term 0.
I20260925 15:18:04.323042 21915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:04.323105 21915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:04.323129 21915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:04.323935 21915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.324038 21915 leader_election.cc:288] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [CANDIDATE]: Term 1 election: Requested vote from peers cf2a7aadb6d449a580be120fbdfb3bcd (127.8.43.190:46255), ab9d2aeed1b743fdbc868ea0877b876e (127.8.43.189:42251)
I20260925 15:18:04.324380 21808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9d2aeed1b743fdbc868ea0877b876e"
I20260925 15:18:04.324451 21808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:04.325234 21808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73755e2ffa1947bcb8bfe37f2e6d7809 in term 1.
I20260925 15:18:04.325440 21858 leader_election.cc:302] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73755e2ffa1947bcb8bfe37f2e6d7809, ab9d2aeed1b743fdbc868ea0877b876e; no voters:
I20260925 15:18:04.325560 21733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd"
I20260925 15:18:04.325634 21733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:04.326750 21733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73755e2ffa1947bcb8bfe37f2e6d7809 in term 1.
I20260925 15:18:04.327082 21915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:04.327261 21915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [term 1 LEADER]: Becoming Leader. State: Replica: 73755e2ffa1947bcb8bfe37f2e6d7809, State: Running, Role: LEADER
I20260925 15:18:04.327342 21915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } }
I20260925 15:18:04.327689 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73755e2ffa1947bcb8bfe37f2e6d7809. Latest consensus state: current_term: 1 leader_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.327764 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:04.328069 21965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:04.328286 21965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:04.330209 21808 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [term 1 FOLLOWER]: Refusing update from remote peer 73755e2ffa1947bcb8bfe37f2e6d7809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:04.330448 21733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [term 1 FOLLOWER]: Refusing update from remote peer 73755e2ffa1947bcb8bfe37f2e6d7809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:04.330837 21915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:04.332554 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73755e2ffa1947bcb8bfe37f2e6d7809. Latest consensus state: current_term: 1 leader_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.332631 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:04.333475 21915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:04.334556 21936 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73755e2ffa1947bcb8bfe37f2e6d7809. Latest consensus state: current_term: 1 leader_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.334643 21936 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:04.336089 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.336160 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:04.336604 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.336686 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:04.336935 21966 mvcc.cc:205] Tried to move back new op lower bound from 7333271487814135808 to 7333271487805304832. Current Snapshot: MvccSnapshot[applied={T|T < 7333271487814135808}]
I20260925 15:18:04.337122 21967 mvcc.cc:205] Tried to move back new op lower bound from 7333271487814135808 to 7333271487805304832. Current Snapshot: MvccSnapshot[applied={T|T < 7333271487814135808}]
I20260925 15:18:04.344025 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.344107 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:04.344573 21936 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2a7aadb6d449a580be120fbdfb3bcd" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46255 } } peers { permanent_uuid: "ab9d2aeed1b743fdbc868ea0877b876e" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 42251 } } peers { permanent_uuid: "73755e2ffa1947bcb8bfe37f2e6d7809" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 41049 } } }
I20260925 15:18:04.344650 21936 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:04.345333 21965 catalog_manager.cc:1347] Generated new cluster ID: f36c0e152c7b4a62a457a37081d053fe
I20260925 15:18:04.345376 21965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:04.368594 21965 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:04.369386 21965 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:04.374095 21965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73755e2ffa1947bcb8bfe37f2e6d7809: Generated new TSK 0
I20260925 15:18:04.374328 21965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:05.272611 21947 catalog_manager.cc:1259] Loaded cluster ID: f36c0e152c7b4a62a457a37081d053fe
I20260925 15:18:05.272691 21947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd: loading cluster ID for follower catalog manager: success
I20260925 15:18:05.273970 21947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd: acquiring CA information for follower catalog manager: success
I20260925 15:18:05.274262 21947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf2a7aadb6d449a580be120fbdfb3bcd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:18:05.316900 21961 catalog_manager.cc:1259] Loaded cluster ID: f36c0e152c7b4a62a457a37081d053fe
I20260925 15:18:05.316978 21961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e: loading cluster ID for follower catalog manager: success
I20260925 15:18:05.317728 21961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e: acquiring CA information for follower catalog manager: success
I20260925 15:18:05.317986 21961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab9d2aeed1b743fdbc868ea0877b876e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:18:05.427487 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 21691
I20260925 15:18:05.432544 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 21766
I20260925 15:18:05.437273 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 21841
2026-09-25T15:18:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2832 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:05 dist-test-slave-c5d6 krb5kdc[21981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:05 dist-test-slave-c5d6 krb5kdc[21981](info): set up 2 sockets
Sep 25 15:18:05 dist-test-slave-c5d6 krb5kdc[21981](info): commencing operation
krb5kdc: starting...
W20260925 15:18:06.480017 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:18:06 dist-test-slave-c5d6 krb5kdc[21981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:18:06.598054 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:42859
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:42859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:06.680641 21994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:06.680886 21994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:06.680924 21994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:06.682060 21994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:06.682132 21994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:06.682160 21994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:06.682191 21994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:06.682216 21994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:06.683714 21994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:42859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:42859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:06.683990 21994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:06.684237 21994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:18:06.689167 21994 server_base.cc:1034] running on GCE node
W20260925 15:18:06.689213 22002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:06.689177 22000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:06.689208 21999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:06.689631 21994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:06.689886 21994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:06.691056 21994 hybrid_clock.cc:648] HybridClock initialized: now 1790349486691045 us; error 43 us; skew 500 ppm
Sep 25 15:18:06 dist-test-slave-c5d6 krb5kdc[21981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349486, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:06.693549 21994 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:18:06.693994 21994 webserver.cc:460] Webserver started at http://127.8.43.190:32953/ using document root <none> and password file <none>
I20260925 15:18:06.694195 21994 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:06.694235 21994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:06.694324 21994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:06.695405 21994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac"
format_stamp: "Formatted at 2026-09-25 15:18:06 on dist-test-slave-c5d6"
server_key: "c8d064283acb973017c119b44e99f587"
server_key_iv: "d3c2f4fff2c4464b07ed7ec81c932dae"
server_key_version: "encryptionkey@0"
I20260925 15:18:06.695711 21994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac"
format_stamp: "Formatted at 2026-09-25 15:18:06 on dist-test-slave-c5d6"
server_key: "c8d064283acb973017c119b44e99f587"
server_key_iv: "d3c2f4fff2c4464b07ed7ec81c932dae"
server_key_version: "encryptionkey@0"
I20260925 15:18:06.696972 21994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:18:06.697691 22009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:06.697912 21994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:18:06.698007 21994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac"
format_stamp: "Formatted at 2026-09-25 15:18:06 on dist-test-slave-c5d6"
server_key: "c8d064283acb973017c119b44e99f587"
server_key_iv: "d3c2f4fff2c4464b07ed7ec81c932dae"
server_key_version: "encryptionkey@0"
I20260925 15:18:06.698078 21994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:06.712548 21994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:06.714414 21994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:06.714519 21994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:06.718395 21994 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:42859
I20260925 15:18:06.718502 22061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:42859 every 8 connection(s)
I20260925 15:18:06.718799 21994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:06.719700 22062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:06.721979 22062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac: Bootstrap starting.
I20260925 15:18:06.722541 22062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:06.722818 22062 log.cc:824] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:06.723026 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 21994
I20260925 15:18:06.723106 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:18:06.723369 8366 external_mini_cluster.cc:1428] Setting key e2fa4e0210e1bd1a3deb339e64b3dfad
I20260925 15:18:06.723599 22062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac: No bootstrap required, opened a new log
I20260925 15:18:06.725325 22062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42859 } }
I20260925 15:18:06.725484 22062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:06.725549 22062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b37c802ae4f4ed3b38f57cda6d4f4ac, State: Initialized, Role: FOLLOWER
I20260925 15:18:06.725682 22062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42859 } }
I20260925 15:18:06.725780 22062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:06.725824 22062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:06.725873 22062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:06.726719 22062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42859 } }
I20260925 15:18:06.726841 22062 leader_election.cc:302] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b37c802ae4f4ed3b38f57cda6d4f4ac; no voters:
I20260925 15:18:06.727015 22062 leader_election.cc:288] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:06.727037 22067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:06.727144 22067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [term 1 LEADER]: Becoming Leader. State: Replica: 8b37c802ae4f4ed3b38f57cda6d4f4ac, State: Running, Role: LEADER
I20260925 15:18:06.727291 22062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:06.727306 22067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42859 } }
I20260925 15:18:06.727737 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b37c802ae4f4ed3b38f57cda6d4f4ac. Latest consensus state: current_term: 1 leader_uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42859 } } }
I20260925 15:18:06.727826 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:06.728348 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b37c802ae4f4ed3b38f57cda6d4f4ac" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42859 } } }
I20260925 15:18:06.728489 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac [sys.catalog]: This master's current role is: LEADER
W20260925 15:18:06.729096 22083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:18:06.729167 22083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:18:06.729247 22074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:06.729420 22074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:06.730967 22074 catalog_manager.cc:1347] Generated new cluster ID: 1318db566080403792bdba0d21cb84bc
I20260925 15:18:06.731015 22074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:18:06 dist-test-slave-c5d6 krb5kdc[21981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:06.735785 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:06.723857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.8.43.190:42859)
0925 15:18:06.724058 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:06.724068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:06.724098 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:18:06.725310 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:06.725317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:06.725362 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:06.725499 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:06.726258 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:06.727387 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:06.728526 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:06.729127 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:06.729246 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:06.732163 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:06.732166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:06.732167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:06.732179 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:06.734083 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:06.734488 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:06.734491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:06.734491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:06.734615 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:06.734854 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:06.734855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:06.734855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:06.735150 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:06.735314 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:06.735445 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:18:06.735574 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260925 15:18:06.736815 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:06.740746 22074 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:06.741474 22074 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:06.750000 22074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b37c802ae4f4ed3b38f57cda6d4f4ac: Generated new TSK 0
I20260925 15:18:06.750248 22074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:06.790987 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 21994
2026-09-25T15:18:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1352 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:06 dist-test-slave-c5d6 krb5kdc[22090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:06 dist-test-slave-c5d6 krb5kdc[22090](info): set up 2 sockets
Sep 25 15:18:06 dist-test-slave-c5d6 krb5kdc[22090](info): commencing operation
krb5kdc: starting...
W20260925 15:18:07.843535 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:18:07 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:18:07.964004 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44259
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:08.045797 22103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:08.045987 22103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:08.046010 22103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:08.047138 22103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:08.047179 22103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:08.047194 22103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:08.047205 22103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:08.047276 22103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:08.048830 22103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:08.049088 22103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:08.049307 22103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:08.053998 22108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.054003 22111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.054066 22103 server_base.cc:1034] running on GCE node
W20260925 15:18:08.054021 22109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.054504 22103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:08.054759 22103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:08.055896 22103 hybrid_clock.cc:648] HybridClock initialized: now 1790349488055876 us; error 38 us; skew 500 ppm
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:08.058395 22103 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:18:08.058826 22103 webserver.cc:460] Webserver started at http://127.8.43.190:34039/ using document root <none> and password file <none>
I20260925 15:18:08.059051 22103 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:08.059118 22103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:08.059296 22103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:08.060319 22103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "2b935b5357f24cb69ab957fa3470c3fe"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "cd4de65c80bff66233a4790b2999e4f7"
server_key_iv: "eb5f9f0b8dbb007e713360a85f974eee"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.060681 22103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "2b935b5357f24cb69ab957fa3470c3fe"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "cd4de65c80bff66233a4790b2999e4f7"
server_key_iv: "eb5f9f0b8dbb007e713360a85f974eee"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.061982 22103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:18:08.062687 22118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.062860 22103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:18:08.062958 22103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "2b935b5357f24cb69ab957fa3470c3fe"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "cd4de65c80bff66233a4790b2999e4f7"
server_key_iv: "eb5f9f0b8dbb007e713360a85f974eee"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.063042 22103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:08.085014 22103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:08.087388 22103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:08.087555 22103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:08.091468 22103 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44259
I20260925 15:18:08.091513 22170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44259 every 8 connection(s)
I20260925 15:18:08.091887 22103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:08.093092 22171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:08.095760 22171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Bootstrap starting.
I20260925 15:18:08.096415 22171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:08.096746 22171 log.cc:824] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:08.097512 22171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: No bootstrap required, opened a new log
I20260925 15:18:08.098594 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 22103
I20260925 15:18:08.098718 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:18:08.098934 8366 external_mini_cluster.cc:1428] Setting key e767cc76aa95dc48198e532103b3cedd
I20260925 15:18:08.099695 22171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:08.099889 22171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:08.099947 22171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b935b5357f24cb69ab957fa3470c3fe, State: Initialized, Role: FOLLOWER
I20260925 15:18:08.100103 22171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:08.100212 22171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:08.100306 22171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:08.100391 22171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:08.101204 22171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:08.101310 22171 leader_election.cc:302] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b935b5357f24cb69ab957fa3470c3fe; no voters:
I20260925 15:18:08.101536 22171 leader_election.cc:288] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:08.101562 22176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:08.101686 22176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 1 LEADER]: Becoming Leader. State: Replica: 2b935b5357f24cb69ab957fa3470c3fe, State: Running, Role: LEADER
I20260925 15:18:08.101824 22171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:08.101840 22176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:08.102396 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b935b5357f24cb69ab957fa3470c3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } } }
I20260925 15:18:08.102694 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:08.103083 22185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:08.103762 22185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:08.102600 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b935b5357f24cb69ab957fa3470c3fe. Latest consensus state: current_term: 1 leader_uuid: "2b935b5357f24cb69ab957fa3470c3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } } }
I20260925 15:18:08.104665 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: This master's current role is: LEADER
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:08.107048 22185 catalog_manager.cc:1347] Generated new cluster ID: 2dc7275813fb42288605e1684a8cfc2a
I20260925 15:18:08.107134 22185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:08.111013 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.099492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.8.43.190:44259)
0925 15:18:08.099663 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.099675 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.099699 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:18:08.099803 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.099808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.099854 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.100008 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:08.100964 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.102103 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.103140 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.103784 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.103887 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.107021 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:08.107026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:08.107026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:08.107039 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:08.109209 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.109654 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.109657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.109657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.109777 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.110004 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.110005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.110006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.110307 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:08.110464 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.110583 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.110745 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260925 15:18:08.116386 22185 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:08.117054 22185 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:08.124583 22185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Generated new TSK 0
I20260925 15:18:08.124748 22185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:18:08.153973 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44259
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:08.244812 22197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:08.245057 22197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:08.245105 22197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:08.246382 22197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:08.246459 22197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:08.246510 22197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:18:08.248210 22197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:08.248498 22197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:08.248744 22197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:08.254139 22203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.254244 22197 server_base.cc:1034] running on GCE node
W20260925 15:18:08.254140 22202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.254179 22205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.254635 22197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:08.254916 22197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:08.256108 22197 hybrid_clock.cc:648] HybridClock initialized: now 1790349488256097 us; error 43 us; skew 500 ppm
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:08.258917 22197 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:18:08.259440 22197 webserver.cc:460] Webserver started at http://127.8.43.129:44973/ using document root <none> and password file <none>
I20260925 15:18:08.259676 22197 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:08.259720 22197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:08.259860 22197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:08.260886 22197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "a8b8b2fff5f343b58c65f000d1e139c5"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "a45bcc6dece7b8c6f40b94a759a00a7f"
server_key_iv: "b6f718a61183bce168c6ffa0d3e54257"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.261246 22197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "a8b8b2fff5f343b58c65f000d1e139c5"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "a45bcc6dece7b8c6f40b94a759a00a7f"
server_key_iv: "b6f718a61183bce168c6ffa0d3e54257"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.262564 22197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:18:08.263306 22212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.263502 22197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:18:08.263589 22197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "a8b8b2fff5f343b58c65f000d1e139c5"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "a45bcc6dece7b8c6f40b94a759a00a7f"
server_key_iv: "b6f718a61183bce168c6ffa0d3e54257"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.263661 22197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:08.282226 22197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:08.284654 22197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:08.284806 22197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:08.285063 22197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:08.285382 22197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:08.285415 22197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.285462 22197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:08.285492 22197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.291771 22197 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:44727
I20260925 15:18:08.291819 22325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:44727 every 8 connection(s)
I20260925 15:18:08.292133 22197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:08.299399 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 22197
I20260925 15:18:08.299530 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:18:08.299783 8366 external_mini_cluster.cc:1428] Setting key 8e71e647c6cd92ecde21be8d738a2055
I20260925 15:18:08.301807 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.292888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:39401 (local address 127.8.43.190:44259)
0925 15:18:08.292947 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.292964 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.293133 (+ 169us) server_negotiation.cc:407] Connection header received
0925 15:18:08.293701 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.293702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.293716 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.293783 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:08.294674 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.295173 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.296160 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.296362 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.296414 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.298477 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:08.298478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:08.298478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:08.298483 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:08.300165 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.300682 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.300683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.300684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.300756 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.301052 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.301054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.301054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.301187 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:08.301292 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.301602 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.301699 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:18:08.302120 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.292840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:39401)
0925 15:18:08.293002 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.293045 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:08.293611 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:08.293842 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:08.293845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:08.294403 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:08.294595 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.294599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.295304 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.295305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.296084 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.296085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.296434 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.296435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.296530 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.297075 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:08.297084 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:08.298398 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:08.300282 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:08.300291 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:08.300293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:08.300569 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:08.300838 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:08.300839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:08.300840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:08.300968 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:08.301284 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:08.301291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:08.301447 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 15:18:08.301577 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:18:08.301752 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260925 15:18:08.302634 22326 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44259
I20260925 15:18:08.302779 22326 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:08.303030 22326 heartbeater.cc:511] Master 127.8.43.190:44259 requested a full tablet report, sending...
I20260925 15:18:08.303817 22135 ts_manager.cc:194] Registered new tserver with Master: a8b8b2fff5f343b58c65f000d1e139c5 (127.8.43.129:44727)
I20260925 15:18:08.304565 22135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:39401
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:18:08.324215 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44259
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:08.404871 22331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:08.405109 22331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:08.405148 22331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:08.406286 22331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:08.406352 22331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:08.406396 22331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:18:08.407866 22331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:08.408134 22331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:08.408377 22331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:08.413451 22339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.413522 22331 server_base.cc:1034] running on GCE node
W20260925 15:18:08.413437 22336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.413436 22337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.413874 22331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:08.414163 22331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:08.415295 22331 hybrid_clock.cc:648] HybridClock initialized: now 1790349488415280 us; error 24 us; skew 500 ppm
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:08.417788 22331 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:18:08.418278 22331 webserver.cc:460] Webserver started at http://127.8.43.130:46157/ using document root <none> and password file <none>
I20260925 15:18:08.418516 22331 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:08.418642 22331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:08.418805 22331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:08.419900 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "90bac09511cb4962b9f389494ef97f67"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "be47a0806bddab07394463fa0fbf26ac"
server_key_iv: "4e4e1545aedb31e4540859b6a5dec7d7"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.420260 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "90bac09511cb4962b9f389494ef97f67"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "be47a0806bddab07394463fa0fbf26ac"
server_key_iv: "4e4e1545aedb31e4540859b6a5dec7d7"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.421583 22331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:18:08.422323 22346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.422544 22331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:08.422648 22331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "90bac09511cb4962b9f389494ef97f67"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "be47a0806bddab07394463fa0fbf26ac"
server_key_iv: "4e4e1545aedb31e4540859b6a5dec7d7"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.422719 22331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:08.436961 22331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:08.438870 22331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:08.438985 22331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:08.439162 22331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:08.439515 22331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:08.439551 22331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.439595 22331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:08.439621 22331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.445178 22331 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:34569
I20260925 15:18:08.445250 22459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:34569 every 8 connection(s)
I20260925 15:18:08.445561 22331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:08.448676 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 22331
I20260925 15:18:08.448770 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:18:08.448946 8366 external_mini_cluster.cc:1428] Setting key 946d8aaa41f7812d136e49d025950c86
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:08.456089 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.446289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:46781 (local address 127.8.43.190:44259)
0925 15:18:08.446339 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.446341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.446574 (+ 233us) server_negotiation.cc:407] Connection header received
0925 15:18:08.447117 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.447119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.447129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.447189 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:08.448168 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.448846 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.450046 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.450389 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.450459 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.452946 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:08.452947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:08.452948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:08.452957 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:08.454533 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.455003 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.455004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.455005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.455074 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.455406 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.455407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.455408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.455534 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:08.455617 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.455894 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.455982 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:18:08.456288 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.446281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:46781)
0925 15:18:08.446450 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.446491 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:08.446946 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:08.447318 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:08.447322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:08.447896 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:08.448082 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.448086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.448931 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.448932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.449965 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.449968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.450492 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.450493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.450593 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.451184 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:08.451192 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:08.452816 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:08.454642 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:08.454649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:08.454651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:08.454898 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:08.455149 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:08.455150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:08.455151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:08.455331 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:08.455607 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:08.455614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:08.455755 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:18:08.455859 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:18:08.456037 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
I20260925 15:18:08.456899 22460 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44259
I20260925 15:18:08.457011 22460 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:08.457188 22460 heartbeater.cc:511] Master 127.8.43.190:44259 requested a full tablet report, sending...
I20260925 15:18:08.457607 22135 ts_manager.cc:194] Registered new tserver with Master: 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
I20260925 15:18:08.458051 22135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:46781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:18:08.473681 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44259
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:08.553865 22465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:08.554064 22465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:08.554085 22465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:08.555303 22465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:08.555344 22465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:08.555371 22465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:18:08.556787 22465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:08.557013 22465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:08.557227 22465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:08.562129 22471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.562135 22470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.562135 22473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.562279 22465 server_base.cc:1034] running on GCE node
I20260925 15:18:08.562556 22465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:08.562760 22465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:08.563928 22465 hybrid_clock.cc:648] HybridClock initialized: now 1790349488563915 us; error 26 us; skew 500 ppm
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:08.566424 22465 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:18:08.566956 22465 webserver.cc:460] Webserver started at http://127.8.43.131:45859/ using document root <none> and password file <none>
I20260925 15:18:08.567165 22465 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:08.567209 22465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:08.567373 22465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:08.568360 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "106c7c50ad1a4514bd424d7843ec2e88"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "4a5d0bccca7797987281973866d20f00"
server_key_iv: "f685846fe48c6ef5e85e92e29ed24e13"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.568732 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "106c7c50ad1a4514bd424d7843ec2e88"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "4a5d0bccca7797987281973866d20f00"
server_key_iv: "f685846fe48c6ef5e85e92e29ed24e13"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.569988 22465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:18:08.570700 22480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.570888 22465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:18:08.570973 22465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "106c7c50ad1a4514bd424d7843ec2e88"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "4a5d0bccca7797987281973866d20f00"
server_key_iv: "f685846fe48c6ef5e85e92e29ed24e13"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.571041 22465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:08.580755 22465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:08.582880 22465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:08.583032 22465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:08.583289 22465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:08.583599 22465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:08.583632 22465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.583673 22465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:08.583699 22465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.589628 22465 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:41877
I20260925 15:18:08.589668 22593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:41877 every 8 connection(s)
I20260925 15:18:08.589994 22465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:08.598241 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 22465
I20260925 15:18:08.598361 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:18:08.598536 8366 external_mini_cluster.cc:1428] Setting key 607721e6e05dbdb258abbd124cf8252a
I20260925 15:18:08.599633 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.590745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:33367 (local address 127.8.43.190:44259)
0925 15:18:08.590815 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.590817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.590978 (+ 161us) server_negotiation.cc:407] Connection header received
0925 15:18:08.591611 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.591612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.591623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.591688 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:08.592546 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.593076 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.594133 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.594334 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.594419 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.596510 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:08.596512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:08.596513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:08.596520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:08.597942 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.598393 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.598395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.598396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.598458 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.598794 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.598795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.598796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.598981 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:08.599056 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.599420 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.599525 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:18:08.599779 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.590695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:33367)
0925 15:18:08.590846 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.590887 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:08.591465 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:08.591744 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:08.591747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:08.592272 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:08.592470 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.592475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.593150 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.593151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.594043 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.594046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.594459 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.594460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.594581 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.595076 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:08.595084 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:08.596416 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:08.598062 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:08.598068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:08.598070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:08.598302 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:08.598536 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:08.598536 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:08.598537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:08.598642 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:08.599059 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:08.599065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:08.599267 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 15:18:08.599383 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:18:08.599563 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260925 15:18:08.600212 22594 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44259
I20260925 15:18:08.600310 22594 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:08.600484 22594 heartbeater.cc:511] Master 127.8.43.190:44259 requested a full tablet report, sending...
I20260925 15:18:08.600863 22135 ts_manager.cc:194] Registered new tserver with Master: 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877)
I20260925 15:18:08.601205 22135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:33367
I20260925 15:18:08.610422 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:18:08.618278 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.611935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.8.43.190:44259)
0925 15:18:08.611987 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.611989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.612079 (+ 90us) server_negotiation.cc:407] Connection header received
0925 15:18:08.612136 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.612136 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.612146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.612203 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:08.612722 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.613619 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.614459 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.614652 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.614710 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.615950 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:08.615951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:08.615951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:08.615957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:08.617236 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.617559 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.617560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.617560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.617616 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.617778 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.617778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.617779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.617918 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:08.618007 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.618086 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.618160 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:18:08.619938 22135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:08.621644 22135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:08.628278 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.624084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44727 (local address 127.0.0.1:37516)
0925 15:18:08.624296 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.624304 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:08.624343 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:08.625058 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:08.625059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:08.625062 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:08.625112 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.625113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.626345 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.626346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.627172 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.627174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.628010 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.628011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.628090 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.628105 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:18:08.628149 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:18:08.628198 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":29,"thread_start_us":51,"threads_started":1}
I20260925 15:18:08.628645 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.624312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.8.43.129:44727)
0925 15:18:08.624877 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.624881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.624891 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:08.624930 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.624932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.624948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.625054 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:08.625177 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.626233 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.627292 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.627911 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.628407 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.628500 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.628525 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.628568 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":111,"threads_started":1}
I20260925 15:18:08.628705 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.624067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:34569 (local address 127.0.0.1:59852)
0925 15:18:08.624230 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.624270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:08.624337 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:08.624727 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:08.624730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:08.624732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:08.624795 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.624798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.626133 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.626134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.627575 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.627578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.628482 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.628483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.628556 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.628559 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:18:08.628590 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 15:18:08.628628 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
I20260925 15:18:08.629055 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.624332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.8.43.130:34569)
0925 15:18:08.624539 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.624543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.624552 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:18:08.624584 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.624586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.624598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.624710 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:08.624881 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.626035 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.627670 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.628391 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.628869 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.628941 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.628969 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.629000 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260925 15:18:08.629496 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.624067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41877 (local address 127.0.0.1:48340)
0925 15:18:08.624243 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.624412 (+ 169us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:08.624447 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:08.625201 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:08.625201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:08.625203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:08.625241 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.625242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.626460 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.626461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.627874 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.627876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.629026 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.629027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.629093 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.629099 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:18:08.629357 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 15:18:08.629404 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260925 15:18:08.629573 22260 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2e50a20b2746d196774ced1f1c1646 (DEFAULT_TABLE table=test-table [id=10cb320949bd457ab1d24ae4d06bb0ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:08.629797 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.624567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.8.43.131:41877)
0925 15:18:08.624815 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.624820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.624832 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:18:08.624874 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.624877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.625016 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.625111 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:08.625358 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.626335 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.628002 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.628801 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.628877 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.628983 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.629667 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.629711 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":126,"threads_started":1}
I20260925 15:18:08.629733 22394 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2e50a20b2746d196774ced1f1c1646 (DEFAULT_TABLE table=test-table [id=10cb320949bd457ab1d24ae4d06bb0ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:08.629983 22260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2e50a20b2746d196774ced1f1c1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:08.630106 22394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2e50a20b2746d196774ced1f1c1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:08.630410 22528 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2e50a20b2746d196774ced1f1c1646 (DEFAULT_TABLE table=test-table [id=10cb320949bd457ab1d24ae4d06bb0ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:08.630702 22528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c2e50a20b2746d196774ced1f1c1646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:08.632755 22608 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Bootstrap starting.
I20260925 15:18:08.633455 22609 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Bootstrap starting.
I20260925 15:18:08.633661 22608 tablet_bootstrap.cc:654] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:08.634219 22609 tablet_bootstrap.cc:654] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:08.634469 22608 log.cc:824] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:08.634867 22609 log.cc:824] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:08.634886 22610 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: Bootstrap starting.
I20260925 15:18:08.635576 22608 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: No bootstrap required, opened a new log
I20260925 15:18:08.635604 22610 tablet_bootstrap.cc:654] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:08.635659 22608 ts_tablet_manager.cc:1397] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:08.636179 22610 log.cc:824] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:08.636864 22609 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: No bootstrap required, opened a new log
I20260925 15:18:08.636942 22609 ts_tablet_manager.cc:1397] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:18:08.637091 22610 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: No bootstrap required, opened a new log
I20260925 15:18:08.637182 22610 ts_tablet_manager.cc:1397] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:18:08.638003 22608 raft_consensus.cc:348] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.638217 22608 raft_consensus.cc:374] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:08.638258 22608 raft_consensus.cc:729] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b8b2fff5f343b58c65f000d1e139c5, State: Initialized, Role: FOLLOWER
I20260925 15:18:08.638643 22608 consensus_queue.cc:260] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.638990 22608 ts_tablet_manager.cc:1428] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:18:08.638896 22610 raft_consensus.cc:348] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.638994 22326 heartbeater.cc:503] Master 127.8.43.190:44259 was elected leader, sending a full tablet report...
I20260925 15:18:08.639099 22610 raft_consensus.cc:374] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:08.639164 22610 raft_consensus.cc:729] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106c7c50ad1a4514bd424d7843ec2e88, State: Initialized, Role: FOLLOWER
I20260925 15:18:08.639315 22610 consensus_queue.cc:260] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.639290 22609 raft_consensus.cc:348] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.639530 22609 raft_consensus.cc:374] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:08.639607 22609 raft_consensus.cc:729] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bac09511cb4962b9f389494ef97f67, State: Initialized, Role: FOLLOWER
I20260925 15:18:08.639644 22610 ts_tablet_manager.cc:1428] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 15:18:08.639665 22594 heartbeater.cc:503] Master 127.8.43.190:44259 was elected leader, sending a full tablet report...
I20260925 15:18:08.639782 22609 consensus_queue.cc:260] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.640093 22609 ts_tablet_manager.cc:1428] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:08.640237 22460 heartbeater.cc:503] Master 127.8.43.190:44259 was elected leader, sending a full tablet report...
W20260925 15:18:08.696118 22461 tablet.cc:2369] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:18:08.735177 22614 raft_consensus.cc:482] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:08.735340 22614 raft_consensus.cc:504] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.735982 22614 leader_election.cc:288] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
I20260925 15:18:08.740356 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.736334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:41991 (local address 127.8.43.131:41877)
0925 15:18:08.736397 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.736400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.736410 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:08.736441 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.736442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.736452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.736530 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:08.736745 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.737569 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.739239 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.739890 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.739988 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.739998 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.740236 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.740279 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:18:08.740356 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.736305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:60675 (local address 127.8.43.130:34569)
0925 15:18:08.736353 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.736354 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.736615 (+ 261us) server_negotiation.cc:407] Connection header received
0925 15:18:08.736643 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.736644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.736656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.736723 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:08.736879 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.737632 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.739241 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.739889 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.739988 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.739998 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.740236 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.740279 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:18:08.740365 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.736209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41877 (local address 127.8.43.129:41991)
0925 15:18:08.736276 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.736291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:08.736360 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:08.736555 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:08.736556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:08.736558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:08.736639 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.736641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.737732 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.737734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.739098 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.739101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.740013 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.740014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.740135 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.740141 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:18:08.740212 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:18:08.740266 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":53}
I20260925 15:18:08.740371 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.736209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:34569 (local address 127.8.43.129:60675)
0925 15:18:08.736478 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.736487 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:08.736565 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:08.736739 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:08.736740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:08.736741 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:08.736804 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.736805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.737732 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.737734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.739098 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:08.739101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.740013 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.740014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:08.740135 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.740142 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:18:08.740212 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:18:08.740265 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":135,"thread_start_us":161,"threads_started":1}
I20260925 15:18:08.740736 22548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2e50a20b2746d196774ced1f1c1646" candidate_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106c7c50ad1a4514bd424d7843ec2e88" is_pre_election: true
I20260925 15:18:08.740803 22414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2e50a20b2746d196774ced1f1c1646" candidate_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bac09511cb4962b9f389494ef97f67" is_pre_election: true
I20260925 15:18:08.740888 22548 raft_consensus.cc:2454] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8b8b2fff5f343b58c65f000d1e139c5 in term 0.
I20260925 15:18:08.740993 22414 raft_consensus.cc:2454] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8b8b2fff5f343b58c65f000d1e139c5 in term 0.
I20260925 15:18:08.741137 22214 leader_election.cc:302] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106c7c50ad1a4514bd424d7843ec2e88, a8b8b2fff5f343b58c65f000d1e139c5; no voters:
I20260925 15:18:08.741279 22614 raft_consensus.cc:2781] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:08.741340 22614 raft_consensus.cc:482] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:08.741374 22614 raft_consensus.cc:3037] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:08.742113 22614 raft_consensus.cc:504] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.742265 22614 leader_election.cc:288] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 1 election: Requested vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
I20260925 15:18:08.742462 22414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2e50a20b2746d196774ced1f1c1646" candidate_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bac09511cb4962b9f389494ef97f67"
I20260925 15:18:08.742511 22548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2e50a20b2746d196774ced1f1c1646" candidate_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106c7c50ad1a4514bd424d7843ec2e88"
I20260925 15:18:08.742560 22414 raft_consensus.cc:3037] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:08.742602 22548 raft_consensus.cc:3037] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:08.743348 22414 raft_consensus.cc:2454] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8b8b2fff5f343b58c65f000d1e139c5 in term 1.
I20260925 15:18:08.743567 22548 raft_consensus.cc:2454] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8b8b2fff5f343b58c65f000d1e139c5 in term 1.
I20260925 15:18:08.743671 22213 leader_election.cc:302] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90bac09511cb4962b9f389494ef97f67, a8b8b2fff5f343b58c65f000d1e139c5; no voters:
I20260925 15:18:08.743762 22614 raft_consensus.cc:2781] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:08.743927 22614 raft_consensus.cc:686] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 LEADER]: Becoming Leader. State: Replica: a8b8b2fff5f343b58c65f000d1e139c5, State: Running, Role: LEADER
I20260925 15:18:08.744040 22614 consensus_queue.cc:237] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:08.744915 22135 catalog_manager.cc:5495] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8b8b2fff5f343b58c65f000d1e139c5 (127.8.43.129). New cstate: current_term: 1 leader_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } health_report { overall_health: HEALTHY } } }
I20260925 15:18:08.779100 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.776523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.8.43.129:44727)
0925 15:18:08.776601 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.776603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.776613 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:08.776662 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.776663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.776683 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.776775 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:08.776940 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.777411 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.778280 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.778486 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.778562 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.778744 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:08.778850 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:08.778947 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.778965 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.778985 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:18:08.781241 22548 raft_consensus.cc:1261] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 1 FOLLOWER]: Refusing update from remote peer a8b8b2fff5f343b58c65f000d1e139c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:08.781435 22414 raft_consensus.cc:1261] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 1 FOLLOWER]: Refusing update from remote peer a8b8b2fff5f343b58c65f000d1e139c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:08.781765 22618 consensus_queue.cc:1035] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:08.781965 22614 consensus_queue.cc:1035] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:08.786738 22135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:08.786846 22135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:08.787604 22135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Sending DeleteTablet for 3 replicas of tablet 3c2e50a20b2746d196774ced1f1c1646
I20260925 15:18:08.787916 22528 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2e50a20b2746d196774ced1f1c1646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:08 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:48340
I20260925 15:18:08.787926 22260 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2e50a20b2746d196774ced1f1c1646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:08 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:37516
I20260925 15:18:08.788064 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 22197
I20260925 15:18:08.788130 22394 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2e50a20b2746d196774ced1f1c1646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:08 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:59852
I20260925 15:18:08.788830 22633 tablet_replica.cc:330] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: stopping tablet replica
I20260925 15:18:08.788992 22633 raft_consensus.cc:2229] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:08.789145 22633 raft_consensus.cc:2258] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:08.789425 22631 tablet_replica.cc:330] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: stopping tablet replica
I20260925 15:18:08.789721 22633 ts_tablet_manager.cc:1905] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:08.790922 22631 raft_consensus.cc:2229] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:08.791064 22631 raft_consensus.cc:2258] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:08.791272 22633 ts_tablet_manager.cc:1918] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:08.791353 22633 log.cc:1196] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/3c2e50a20b2746d196774ced1f1c1646
I20260925 15:18:08.791558 22631 ts_tablet_manager.cc:1905] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:08.791602 22633 ts_tablet_manager.cc:1939] T 3c2e50a20b2746d196774ced1f1c1646 P 90bac09511cb4962b9f389494ef97f67: Deleting consensus metadata
I20260925 15:18:08.792003 22119 catalog_manager.cc:4841] TS 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569): tablet 3c2e50a20b2746d196774ced1f1c1646 (table test-table [id=10cb320949bd457ab1d24ae4d06bb0ae]) successfully deleted
I20260925 15:18:08.793031 22631 ts_tablet_manager.cc:1918] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:08.793107 22631 log.cc:1196] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/3c2e50a20b2746d196774ced1f1c1646
I20260925 15:18:08.793377 22631 ts_tablet_manager.cc:1939] T 3c2e50a20b2746d196774ced1f1c1646 P 106c7c50ad1a4514bd424d7843ec2e88: Deleting consensus metadata
W20260925 15:18:08.793583 22122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:44727 (error 108)
I20260925 15:18:08.793779 22120 catalog_manager.cc:4841] TS 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877): tablet 3c2e50a20b2746d196774ced1f1c1646 (table test-table [id=10cb320949bd457ab1d24ae4d06bb0ae]) successfully deleted
I20260925 15:18:08.793877 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 22331
I20260925 15:18:08.795583 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.795413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44727 (local address 127.0.0.1:37538)
0925 15:18:08.795519 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:08.795557 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:44727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 15:18:08.795701 22122 catalog_manager.cc:4584] TS a8b8b2fff5f343b58c65f000d1e139c5 (127.8.43.129:44727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c2e50a20b2746d196774ced1f1c1646: Network error: Client connection negotiation failed: client connection to 127.8.43.129:44727: connect: Connection refused (error 111)
I20260925 15:18:08.799300 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 22465
I20260925 15:18:08.804184 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 22103
I20260925 15:18:08.809427 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44259
--webserver_interface=127.8.43.190
--webserver_port=34039
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:08.890115 22635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:08.890349 22635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:08.890384 22635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:08.891551 22635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:08.891623 22635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:08.891649 22635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:08.891669 22635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:08.891700 22635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:08.893124 22635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=34039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:08.893395 22635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:08.893618 22635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:08.902535 22642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.902661 22644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.902675 22641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.903000 22635 server_base.cc:1034] running on GCE node
I20260925 15:18:08.903354 22635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:08.903776 22635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:08.904978 22635 hybrid_clock.cc:648] HybridClock initialized: now 1790349488904954 us; error 34 us; skew 500 ppm
Sep 25 15:18:08 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:08.911391 22635 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:18:08.911839 22635 webserver.cc:460] Webserver started at http://127.8.43.190:34039/ using document root <none> and password file <none>
I20260925 15:18:08.912072 22635 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:08.912156 22635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:08.913692 22635 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:08.914367 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.914589 22635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:08.914708 22635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "2b935b5357f24cb69ab957fa3470c3fe"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "cd4de65c80bff66233a4790b2999e4f7"
server_key_iv: "eb5f9f0b8dbb007e713360a85f974eee"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.915117 22635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:08.934921 22635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:08.936879 22635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:08.937045 22635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:08.941246 22635 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44259
I20260925 15:18:08.941283 22703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44259 every 8 connection(s)
I20260925 15:18:08.941716 22635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:08.943363 22704 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:18:08.944139 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Bootstrap starting.
I20260925 15:18:08.944154 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 22635
I20260925 15:18:08.944455 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:44727
--local_ip_for_outbound_sockets=127.8.43.129
--tserver_master_addrs=127.8.43.190:44259
--webserver_port=44973
--webserver_interface=127.8.43.129
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:08.952839 22704 log.cc:824] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:08.954681 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:18:08.954980 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Bootstrap complete.
I20260925 15:18:08.957271 22704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:08.957486 22704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b935b5357f24cb69ab957fa3470c3fe, State: Initialized, Role: FOLLOWER
I20260925 15:18:08.957665 22704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:08.957742 22704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:08.957806 22704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:08.957854 22704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:18:08.958897 22704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:08.958984 22704 leader_election.cc:302] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b935b5357f24cb69ab957fa3470c3fe; no voters:
I20260925 15:18:08.959170 22704 leader_election.cc:288] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:18:08.959245 22708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:18:08.959367 22708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 2 LEADER]: Becoming Leader. State: Replica: 2b935b5357f24cb69ab957fa3470c3fe, State: Running, Role: LEADER
I20260925 15:18:08.959451 22704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:08.959458 22708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:08.959738 22708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b935b5357f24cb69ab957fa3470c3fe. Latest consensus state: current_term: 2 leader_uuid: "2b935b5357f24cb69ab957fa3470c3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } } }
I20260925 15:18:08.959856 22708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:08.960054 22709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b935b5357f24cb69ab957fa3470c3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } } }
I20260925 15:18:08.960122 22709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:08.960638 22712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:08.961371 22712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:08.961489 22712 catalog_manager.cc:1259] Loaded cluster ID: 2dc7275813fb42288605e1684a8cfc2a
I20260925 15:18:08.961779 22712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:08.962057 22712 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:18:08.966678 22712 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:18:08.969238 22712 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:08.969436 22712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Loaded TSK: 0
I20260925 15:18:08.969641 22712 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:18:09.030287 22706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:09.030498 22706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:09.030520 22706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:09.031798 22706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:09.031840 22706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:09.031867 22706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:18:09.033301 22706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:44727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=44973
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:09.033533 22706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:09.033792 22706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:09.038762 22734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.038769 22732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.038767 22731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:09.039158 22706 server_base.cc:1034] running on GCE node
I20260925 15:18:09.039476 22706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:09.039746 22706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:09.040936 22706 hybrid_clock.cc:648] HybridClock initialized: now 1790349489040920 us; error 33 us; skew 500 ppm
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:09.043735 22706 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:18:09.044267 22706 webserver.cc:460] Webserver started at http://127.8.43.129:44973/ using document root <none> and password file <none>
I20260925 15:18:09.044528 22706 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:09.044611 22706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:09.046418 22706 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:18:09.047199 22741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.047395 22706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:09.047478 22706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "a8b8b2fff5f343b58c65f000d1e139c5"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "a45bcc6dece7b8c6f40b94a759a00a7f"
server_key_iv: "b6f718a61183bce168c6ffa0d3e54257"
server_key_version: "encryptionkey@0"
I20260925 15:18:09.047842 22706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:09.056365 22706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:09.058478 22706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:09.058624 22706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:09.058851 22706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:09.059310 22748 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:18:09.061249 22706 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:18:09.061317 22706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:18:09.061359 22706 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:18:09.062008 22706 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:18:09.062063 22706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:18:09.062139 22748 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Bootstrap starting.
I20260925 15:18:09.069049 22706 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:44727
I20260925 15:18:09.069087 22855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:44727 every 8 connection(s)
I20260925 15:18:09.069442 22706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:18:09.069788 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 22706
I20260925 15:18:09.070075 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:34569
--local_ip_for_outbound_sockets=127.8.43.130
--tserver_master_addrs=127.8.43.190:44259
--webserver_port=46157
--webserver_interface=127.8.43.130
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:09.073196 22748 log.cc:824] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:09.076772 22748 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:18:09.077078 22748 tablet_bootstrap.cc:492] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Bootstrap complete.
I20260925 15:18:09.078277 22748 ts_tablet_manager.cc:1397] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.009s
I20260925 15:18:09.079849 22748 raft_consensus.cc:348] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.080179 22748 raft_consensus.cc:729] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b8b2fff5f343b58c65f000d1e139c5, State: Initialized, Role: FOLLOWER
I20260925 15:18:09.080349 22748 consensus_queue.cc:260] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.080606 22748 ts_tablet_manager.cc:1428] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:09.086802 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.070189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:36537)
0925 15:18:09.073590 (+ 3401us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.073642 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.074242 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.074863 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:09.074867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:09.075291 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:09.075589 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.075596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.077536 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.077537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.079060 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.079063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.080092 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.080093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.080386 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.081147 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:09.081157 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:09.082846 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:09.085032 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:09.085038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:09.085040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:09.085329 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:09.085603 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:09.085604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:09.085604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:09.085739 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:09.086158 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:09.086165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:09.086315 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 15:18:09.086423 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:18:09.086580 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3354,"thread_start_us":81,"threads_started":1}
I20260925 15:18:09.087105 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.070303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:36537 (local address 127.8.43.190:44259)
0925 15:18:09.073495 (+ 3192us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.073505 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.073783 (+ 278us) server_negotiation.cc:407] Connection header received
0925 15:18:09.074471 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.074478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.074545 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.074704 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:09.075733 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.077409 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.079162 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.079885 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.079994 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.082956 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:09.082960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:09.082961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:09.082972 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:09.084941 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:09.085414 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:09.085417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:09.085417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:09.085524 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:09.085807 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:09.085808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:09.085809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:09.086077 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:09.086158 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.086489 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.086628 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3140,"thread_start_us":88,"threads_started":1}
I20260925 15:18:09.087761 22856 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44259
I20260925 15:18:09.087855 22856 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:09.088030 22856 heartbeater.cc:511] Master 127.8.43.190:44259 requested a full tablet report, sending...
I20260925 15:18:09.088858 22668 ts_manager.cc:194] Registered new tserver with Master: a8b8b2fff5f343b58c65f000d1e139c5 (127.8.43.129:44727)
I20260925 15:18:09.090032 22668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:36537
I20260925 15:18:09.090983 22856 heartbeater.cc:503] Master 127.8.43.190:44259 was elected leader, sending a full tablet report...
I20260925 15:18:09.095677 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.089462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44727 (local address 127.0.0.1:37540)
0925 15:18:09.091448 (+ 1986us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.091483 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.091606 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.091853 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:09.091856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:09.091858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:09.091916 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.091917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.093113 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.093114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.094387 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.094390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.095379 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.095380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.095461 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.095474 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:18:09.095552 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:18:09.095606 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1960,"thread_start_us":84,"threads_started":1}
I20260925 15:18:09.095695 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.089707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.8.43.129:44727)
0925 15:18:09.090327 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.090331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.091546 (+ 1215us) server_negotiation.cc:407] Connection header received
0925 15:18:09.091726 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.091730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.091749 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.091844 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:09.091976 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.093022 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.094472 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.095276 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.095375 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.095426 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.095571 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.095608 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":75,"threads_started":1}
I20260925 15:18:09.096292 22790 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2e50a20b2746d196774ced1f1c1646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:08 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:37540
I20260925 15:18:09.096946 22865 tablet_replica.cc:330] stopping tablet replica
I20260925 15:18:09.097044 22865 raft_consensus.cc:2229] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:09.097103 22865 raft_consensus.cc:2258] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:09.097397 22865 ts_tablet_manager.cc:1905] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:09.099915 22865 ts_tablet_manager.cc:1918] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:09.099996 22865 log.cc:1196] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/3c2e50a20b2746d196774ced1f1c1646
I20260925 15:18:09.100226 22865 ts_tablet_manager.cc:1939] T 3c2e50a20b2746d196774ced1f1c1646 P a8b8b2fff5f343b58c65f000d1e139c5: Deleting consensus metadata
I20260925 15:18:09.100728 22655 catalog_manager.cc:4841] TS a8b8b2fff5f343b58c65f000d1e139c5 (127.8.43.129:44727): tablet 3c2e50a20b2746d196774ced1f1c1646 (table test-table [id=10cb320949bd457ab1d24ae4d06bb0ae]) successfully deleted
W20260925 15:18:09.157505 22859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:09.157745 22859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:09.157786 22859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:09.159021 22859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:09.159086 22859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:09.159132 22859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:18:09.160861 22859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:34569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=46157
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:09.161156 22859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:09.161443 22859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:09.166375 22873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.166400 22870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.166433 22871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:09.166656 22859 server_base.cc:1034] running on GCE node
I20260925 15:18:09.166896 22859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:09.167099 22859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:09.168274 22859 hybrid_clock.cc:648] HybridClock initialized: now 1790349489168254 us; error 32 us; skew 500 ppm
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:09.170684 22859 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:18:09.171089 22859 webserver.cc:460] Webserver started at http://127.8.43.130:46157/ using document root <none> and password file <none>
I20260925 15:18:09.171350 22859 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:09.171408 22859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:09.172763 22859 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:09.173368 22880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.173559 22859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:18:09.173645 22859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "90bac09511cb4962b9f389494ef97f67"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "be47a0806bddab07394463fa0fbf26ac"
server_key_iv: "4e4e1545aedb31e4540859b6a5dec7d7"
server_key_version: "encryptionkey@0"
I20260925 15:18:09.173980 22859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:09.192839 22859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:09.194689 22859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:09.194830 22859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:09.195082 22859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:09.195468 22859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:09.195504 22859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.195531 22859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:09.195576 22859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.201184 22859 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:34569
I20260925 15:18:09.201246 22993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:34569 every 8 connection(s)
I20260925 15:18:09.201611 22859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:09.205265 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 22859
I20260925 15:18:09.205605 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:41877
--local_ip_for_outbound_sockets=127.8.43.131
--tserver_master_addrs=127.8.43.190:44259
--webserver_port=45859
--webserver_interface=127.8.43.131
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:09.211951 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.202323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:58917 (local address 127.8.43.190:44259)
0925 15:18:09.202413 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.202415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.202661 (+ 246us) server_negotiation.cc:407] Connection header received
0925 15:18:09.203125 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.203126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.203138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.203190 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:09.204137 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.204628 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.205710 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.205929 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.206019 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.208538 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:09.208540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:09.208540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:09.208548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:09.210364 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:09.210780 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:09.210781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:09.210782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:09.210849 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:09.211155 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:09.211157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:09.211157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:09.211327 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:09.211401 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.211715 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.211813 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:18:09.212077 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.202304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:58917)
0925 15:18:09.202508 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.202559 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.203040 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.203304 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:09.203308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:09.203860 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:09.204059 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.204064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.204726 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.204727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.205608 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.205611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.206031 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.206032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.206149 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.206698 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:09.206707 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:09.208443 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:09.210458 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:09.210466 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:09.210467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:09.210703 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:09.210927 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:09.210928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:09.210928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:09.211086 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:09.211439 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:09.211445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:09.211565 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 15:18:09.211652 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:18:09.211820 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":113,"threads_started":1}
I20260925 15:18:09.212572 22994 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44259
I20260925 15:18:09.212685 22994 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:09.212867 22994 heartbeater.cc:511] Master 127.8.43.190:44259 requested a full tablet report, sending...
I20260925 15:18:09.213304 22668 ts_manager.cc:194] Registered new tserver with Master: 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
I20260925 15:18:09.213856 22668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:58917
W20260925 15:18:09.288038 22997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:09.288235 22997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:09.288257 22997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:09.289317 22997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:09.289351 22997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:09.289376 22997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:18:09.290726 22997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:41877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=45859
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:09.290906 22997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:09.291086 22997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:09.295787 23002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.295797 23003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.295954 23005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:09.296324 22997 server_base.cc:1034] running on GCE node
I20260925 15:18:09.296532 22997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:09.296757 22997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:09.297928 22997 hybrid_clock.cc:648] HybridClock initialized: now 1790349489297893 us; error 71 us; skew 500 ppm
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:09.300403 22997 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:18:09.300820 22997 webserver.cc:460] Webserver started at http://127.8.43.131:45859/ using document root <none> and password file <none>
I20260925 15:18:09.301054 22997 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:09.301136 22997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:09.302457 22997 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:09.303030 23012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.303198 22997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:18:09.303351 22997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "106c7c50ad1a4514bd424d7843ec2e88"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "4a5d0bccca7797987281973866d20f00"
server_key_iv: "f685846fe48c6ef5e85e92e29ed24e13"
server_key_version: "encryptionkey@0"
I20260925 15:18:09.303681 22997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:09.320410 22997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:09.322340 22997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:09.322525 22997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:09.322769 22997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:09.323091 22997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:09.323150 22997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.323199 22997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:09.323254 22997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.329463 22997 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:41877
I20260925 15:18:09.329526 23125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:41877 every 8 connection(s)
I20260925 15:18:09.329918 22997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:18:09.331058 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 22997
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:09.339303 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.331476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.8.43.190:44259)
0925 15:18:09.331598 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.331600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.331627 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:18:09.331689 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.331690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.331701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.331763 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:09.332566 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.333311 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.334475 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.334678 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.334743 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.336093 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:09.336094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:09.336094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:09.336102 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:09.338049 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:09.338465 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:09.338466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:09.338466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:09.338528 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:09.338736 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:09.338736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:09.338737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:09.338840 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:09.339050 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.339092 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.339177 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260925 15:18:09.340274 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.330644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:44731 (local address 127.8.43.190:44259)
0925 15:18:09.330742 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.330744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.330942 (+ 198us) server_negotiation.cc:407] Connection header received
0925 15:18:09.331617 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.331618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.331632 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.331704 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:09.332865 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.333519 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.334837 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.335007 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.335060 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.337163 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:09.337164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:09.337165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:09.337172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:09.338742 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:09.339120 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:09.339121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:09.339122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:09.339177 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:09.339508 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:09.339509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:09.339509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:09.339643 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:09.339723 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.340089 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.340177 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:18:09.340456 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.330624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:44731)
0925 15:18:09.330815 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.330864 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.331511 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.331775 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:09.331779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:09.332563 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:09.332766 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.332771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.333607 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.333608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.334742 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.334745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.335087 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.335088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.335258 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.335750 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:09.335759 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:09.337066 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:09.338799 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:09.338805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:09.338808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:09.339027 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:09.339280 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:09.339281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:09.339281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:09.339418 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:09.339721 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:09.339727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:09.339905 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 15:18:09.340055 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 15:18:09.340222 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
I20260925 15:18:09.340968 23126 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44259
I20260925 15:18:09.341069 23126 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:09.341238 23126 heartbeater.cc:511] Master 127.8.43.190:44259 requested a full tablet report, sending...
I20260925 15:18:09.341636 22668 ts_manager.cc:194] Registered new tserver with Master: 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877)
I20260925 15:18:09.342010 22668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:44731
I20260925 15:18:09.351079 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:18:09.353957 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.351465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.8.43.190:44259)
0925 15:18:09.351521 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.351523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.351618 (+ 95us) server_negotiation.cc:407] Connection header received
0925 15:18:09.351698 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.351699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.351742 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.351811 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:09.351976 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.352429 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.353193 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.353385 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.353475 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.353649 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:09.353746 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:09.353838 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.353858 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.353875 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:18:09.354225 22668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:09.355949 22668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:09.358088 22790 tablet_service.cc:1467] Processing CreateTablet for tablet 036b80766fa941d1ab86ea5be89968d2 (DEFAULT_TABLE table=test-table [id=0763cafce7c14c0caa8839cba4dfd2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:09.358289 22790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036b80766fa941d1ab86ea5be89968d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:09.360033 22748 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Bootstrap starting.
I20260925 15:18:09.360460 22748 tablet_bootstrap.cc:654] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:09.361631 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.357718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:34569 (local address 127.0.0.1:59866)
0925 15:18:09.357752 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.357762 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.357794 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.358333 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:09.358334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:09.358336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:09.358372 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.358374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.359469 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.359470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.360484 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.360486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.361401 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.361402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.361461 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.361464 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:18:09.361524 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:18:09.361566 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 15:18:09.361649 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.357862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.8.43.130:34569)
0925 15:18:09.358160 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.358184 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.358194 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:09.358236 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.358238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.358253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.358325 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:09.358461 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.359373 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.360567 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.361308 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.361399 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.361468 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.361537 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.361574 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":138,"threads_started":1}
I20260925 15:18:09.362568 22748 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: No bootstrap required, opened a new log
I20260925 15:18:09.362623 22748 ts_tablet_manager.cc:1397] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260925 15:18:09.362779 22748 raft_consensus.cc:348] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.362852 22748 raft_consensus.cc:374] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:09.362913 22748 raft_consensus.cc:729] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b8b2fff5f343b58c65f000d1e139c5, State: Initialized, Role: FOLLOWER
I20260925 15:18:09.362851 22928 tablet_service.cc:1467] Processing CreateTablet for tablet 036b80766fa941d1ab86ea5be89968d2 (DEFAULT_TABLE table=test-table [id=0763cafce7c14c0caa8839cba4dfd2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:09.362970 22748 consensus_queue.cc:260] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.363085 22748 ts_tablet_manager.cc:1428] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.363211 22928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036b80766fa941d1ab86ea5be89968d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:09.363507 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.357720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41877 (local address 127.0.0.1:48348)
0925 15:18:09.357978 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.357987 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.358030 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.359723 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:09.359724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:09.359727 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:09.359770 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.359772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.361210 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.361211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.362119 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.362122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.363252 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.363253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.363324 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.363328 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:18:09.363390 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:18:09.363431 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":11,"thread_start_us":82,"threads_started":1}
I20260925 15:18:09.363719 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.357867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.8.43.131:41877)
0925 15:18:09.359564 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.359568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.359577 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:18:09.359615 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.359617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.359628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.359714 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:09.359828 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.361103 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.362215 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.363095 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.363266 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.363327 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.363458 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.363500 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1665,"thread_start_us":105,"threads_started":1}
I20260925 15:18:09.365990 23136 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Bootstrap starting.
I20260925 15:18:09.366518 23060 tablet_service.cc:1467] Processing CreateTablet for tablet 036b80766fa941d1ab86ea5be89968d2 (DEFAULT_TABLE table=test-table [id=0763cafce7c14c0caa8839cba4dfd2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:09.366612 23136 tablet_bootstrap.cc:654] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:09.366814 23060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036b80766fa941d1ab86ea5be89968d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:09.367158 23136 log.cc:824] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:09.367892 23136 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: No bootstrap required, opened a new log
I20260925 15:18:09.367952 23136 ts_tablet_manager.cc:1397] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:18:09.369531 23138 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: Bootstrap starting.
I20260925 15:18:09.369513 23136 raft_consensus.cc:348] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.369686 23136 raft_consensus.cc:374] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:09.369721 23136 raft_consensus.cc:729] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bac09511cb4962b9f389494ef97f67, State: Initialized, Role: FOLLOWER
I20260925 15:18:09.369848 23136 consensus_queue.cc:260] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.370132 23136 ts_tablet_manager.cc:1428] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:18:09.370175 23138 tablet_bootstrap.cc:654] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:09.370141 22994 heartbeater.cc:503] Master 127.8.43.190:44259 was elected leader, sending a full tablet report...
I20260925 15:18:09.370733 23138 log.cc:824] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:09.371492 23138 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: No bootstrap required, opened a new log
I20260925 15:18:09.371574 23138 ts_tablet_manager.cc:1397] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:18:09.372816 23138 raft_consensus.cc:348] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.372962 23138 raft_consensus.cc:374] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:09.373008 23138 raft_consensus.cc:729] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106c7c50ad1a4514bd424d7843ec2e88, State: Initialized, Role: FOLLOWER
I20260925 15:18:09.373111 23138 consensus_queue.cc:260] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.373354 23138 ts_tablet_manager.cc:1428] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:18:09.373411 23126 heartbeater.cc:503] Master 127.8.43.190:44259 was elected leader, sending a full tablet report...
I20260925 15:18:09.427678 22862 raft_consensus.cc:482] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:09.427802 22862 raft_consensus.cc:504] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.428261 22862 leader_election.cc:288] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
I20260925 15:18:09.431255 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.428687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:55677 (local address 127.8.43.131:41877)
0925 15:18:09.428761 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.428763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.428770 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:18:09.428792 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.428793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.428800 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.428867 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:09.429028 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.429527 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.430483 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.430861 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.430948 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.430957 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.431110 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.431147 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:18:09.431269 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.428710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:38315 (local address 127.8.43.130:34569)
0925 15:18:09.428760 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.428763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.428770 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:18:09.428796 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.428797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.428805 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.428860 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:09.429044 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.429516 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.430506 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.430889 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.430989 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.430997 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.431142 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.431180 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:18:09.431300 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.428422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41877 (local address 127.8.43.129:55677)
0925 15:18:09.428471 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.428484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.428530 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.428872 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:09.428873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:09.428875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:09.428953 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.428955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.429610 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.429612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.430386 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.430388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.430945 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.430946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.431036 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.431039 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:18:09.431105 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:18:09.431149 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":34}
I20260925 15:18:09.431326 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.428454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:34569 (local address 127.8.43.129:38315)
0925 15:18:09.428636 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.428663 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.428717 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.428875 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:09.428876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:09.428878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:09.428970 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.428972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.429602 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.429603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.430390 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:09.430392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.430994 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.430995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:09.431065 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.431068 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:18:09.431136 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:18:09.431176 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":25,"thread_start_us":124,"threads_started":1}
I20260925 15:18:09.431690 22948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036b80766fa941d1ab86ea5be89968d2" candidate_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bac09511cb4962b9f389494ef97f67" is_pre_election: true
I20260925 15:18:09.431702 23080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036b80766fa941d1ab86ea5be89968d2" candidate_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106c7c50ad1a4514bd424d7843ec2e88" is_pre_election: true
I20260925 15:18:09.431840 22948 raft_consensus.cc:2454] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8b8b2fff5f343b58c65f000d1e139c5 in term 0.
I20260925 15:18:09.431857 23080 raft_consensus.cc:2454] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8b8b2fff5f343b58c65f000d1e139c5 in term 0.
I20260925 15:18:09.432063 22743 leader_election.cc:302] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106c7c50ad1a4514bd424d7843ec2e88, a8b8b2fff5f343b58c65f000d1e139c5; no voters:
I20260925 15:18:09.432210 22862 raft_consensus.cc:2781] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:09.432267 22862 raft_consensus.cc:482] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:09.432283 22862 raft_consensus.cc:3037] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:09.433027 22862 raft_consensus.cc:504] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.433162 22862 leader_election.cc:288] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 1 election: Requested vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
I20260925 15:18:09.433297 23080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036b80766fa941d1ab86ea5be89968d2" candidate_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106c7c50ad1a4514bd424d7843ec2e88"
I20260925 15:18:09.433383 23080 raft_consensus.cc:3037] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:09.434120 23080 raft_consensus.cc:2454] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8b8b2fff5f343b58c65f000d1e139c5 in term 1.
I20260925 15:18:09.434283 22743 leader_election.cc:302] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 106c7c50ad1a4514bd424d7843ec2e88, a8b8b2fff5f343b58c65f000d1e139c5; no voters:
I20260925 15:18:09.434288 22948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036b80766fa941d1ab86ea5be89968d2" candidate_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90bac09511cb4962b9f389494ef97f67"
I20260925 15:18:09.434353 22948 raft_consensus.cc:3037] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:09.434444 22862 raft_consensus.cc:2781] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:09.434623 22862 raft_consensus.cc:686] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 LEADER]: Becoming Leader. State: Replica: a8b8b2fff5f343b58c65f000d1e139c5, State: Running, Role: LEADER
I20260925 15:18:09.434716 22862 consensus_queue.cc:237] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.435370 22948 raft_consensus.cc:2454] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8b8b2fff5f343b58c65f000d1e139c5 in term 1.
I20260925 15:18:09.435403 22668 catalog_manager.cc:5495] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8b8b2fff5f343b58c65f000d1e139c5 (127.8.43.129). New cstate: current_term: 1 leader_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } health_report { overall_health: HEALTHY } } }
W20260925 15:18:09.452200 22995 tablet.cc:2369] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:18:09.456007 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.453648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.8.43.129:44727)
0925 15:18:09.453700 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:09.453702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:09.453712 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:09.453786 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:09.453787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:09.453806 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:09.453876 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:09.454002 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.454459 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.455176 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:09.455423 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:09.455503 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:09.455672 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:09.455775 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:09.455858 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:09.455899 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 15:18:09.455918 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:18:09.457495 22948 raft_consensus.cc:1261] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 1 FOLLOWER]: Refusing update from remote peer a8b8b2fff5f343b58c65f000d1e139c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:09.457507 23080 raft_consensus.cc:1261] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 1 FOLLOWER]: Refusing update from remote peer a8b8b2fff5f343b58c65f000d1e139c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:09.457803 22862 consensus_queue.cc:1035] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:09.457957 23143 consensus_queue.cc:1035] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:09.460150 23145 mvcc.cc:205] Tried to move back new op lower bound from 7333271508815167488 to 7333271508725051392. Current Snapshot: MvccSnapshot[applied={T|T < 7333271508815167488}]
I20260925 15:18:09.461193 23147 mvcc.cc:205] Tried to move back new op lower bound from 7333271508815167488 to 7333271508725051392. Current Snapshot: MvccSnapshot[applied={T|T < 7333271508815167488}]
I20260925 15:18:09.462436 22667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:09.462543 22667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:09.463315 22667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Sending DeleteTablet for 3 replicas of tablet 036b80766fa941d1ab86ea5be89968d2
I20260925 15:18:09.463651 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 22706
I20260925 15:18:09.463709 22790 tablet_service.cc:1514] Processing DeleteTablet for tablet 036b80766fa941d1ab86ea5be89968d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:09 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:37540
I20260925 15:18:09.463701 23060 tablet_service.cc:1514] Processing DeleteTablet for tablet 036b80766fa941d1ab86ea5be89968d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:09 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:48348
I20260925 15:18:09.463828 22928 tablet_service.cc:1514] Processing DeleteTablet for tablet 036b80766fa941d1ab86ea5be89968d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:09 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:59866
I20260925 15:18:09.464005 22865 tablet_replica.cc:330] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: stopping tablet replica
I20260925 15:18:09.464061 22865 raft_consensus.cc:2229] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:18:09.464262 23154 tablet_replica.cc:330] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: stopping tablet replica
I20260925 15:18:09.464287 23155 tablet_replica.cc:330] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: stopping tablet replica
I20260925 15:18:09.464357 23154 raft_consensus.cc:2229] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:09.464378 23155 raft_consensus.cc:2229] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:09.464466 23154 raft_consensus.cc:2258] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:09.464491 23155 raft_consensus.cc:2258] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:09.464887 23154 ts_tablet_manager.cc:1905] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:09.464903 23155 ts_tablet_manager.cc:1905] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:09.466367 23154 ts_tablet_manager.cc:1918] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:09.466447 23154 log.cc:1196] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/036b80766fa941d1ab86ea5be89968d2
I20260925 15:18:09.466745 23154 ts_tablet_manager.cc:1939] T 036b80766fa941d1ab86ea5be89968d2 P 106c7c50ad1a4514bd424d7843ec2e88: Deleting consensus metadata
I20260925 15:18:09.466858 23155 ts_tablet_manager.cc:1918] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:09.466931 23155 log.cc:1196] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/036b80766fa941d1ab86ea5be89968d2
I20260925 15:18:09.467154 22653 catalog_manager.cc:4841] TS 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877): tablet 036b80766fa941d1ab86ea5be89968d2 (table test-table [id=0763cafce7c14c0caa8839cba4dfd2fa]) successfully deleted
I20260925 15:18:09.467200 23155 ts_tablet_manager.cc:1939] T 036b80766fa941d1ab86ea5be89968d2 P 90bac09511cb4962b9f389494ef97f67: Deleting consensus metadata
I20260925 15:18:09.467608 22652 catalog_manager.cc:4841] TS 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569): tablet 036b80766fa941d1ab86ea5be89968d2 (table test-table [id=0763cafce7c14c0caa8839cba4dfd2fa]) successfully deleted
W20260925 15:18:09.470592 22655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:44727 (error 108)
I20260925 15:18:09.470755 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 22859
I20260925 15:18:09.473018 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.472889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44727 (local address 127.0.0.1:37560)
0925 15:18:09.472949 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.472991 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:44727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:18:09.473168 22655 catalog_manager.cc:4584] TS a8b8b2fff5f343b58c65f000d1e139c5 (127.8.43.129:44727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 036b80766fa941d1ab86ea5be89968d2: Network error: Client connection negotiation failed: client connection to 127.8.43.129:44727: connect: Connection refused (error 111)
I20260925 15:18:09.476109 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 22997
I20260925 15:18:09.480803 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 22635
I20260925 15:18:09.485824 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44259
--webserver_interface=127.8.43.190
--webserver_port=34039
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:09.565243 23157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:09.565435 23157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:09.565456 23157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:09.566542 23157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:09.566576 23157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:09.566591 23157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:09.566602 23157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:09.566614 23157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:09.568060 23157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=34039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:09.568241 23157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:09.568398 23157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:09.573144 23163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.573167 23162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.573166 23165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:09.573378 23157 server_base.cc:1034] running on GCE node
I20260925 15:18:09.573621 23157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:09.573899 23157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:09.575073 23157 hybrid_clock.cc:648] HybridClock initialized: now 1790349489575065 us; error 31 us; skew 500 ppm
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:09.577559 23157 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:18:09.577973 23157 webserver.cc:460] Webserver started at http://127.8.43.190:34039/ using document root <none> and password file <none>
I20260925 15:18:09.578235 23157 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:09.578305 23157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:09.579679 23157 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:18:09.580379 23172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.580601 23157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:18:09.580703 23157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "2b935b5357f24cb69ab957fa3470c3fe"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "cd4de65c80bff66233a4790b2999e4f7"
server_key_iv: "eb5f9f0b8dbb007e713360a85f974eee"
server_key_version: "encryptionkey@0"
I20260925 15:18:09.581041 23157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:09.597766 23157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:09.600139 23157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:09.600292 23157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:09.604146 23157 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44259
I20260925 15:18:09.604192 23224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44259 every 8 connection(s)
I20260925 15:18:09.604552 23157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:09.606076 23225 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:18:09.606724 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Bootstrap starting.
I20260925 15:18:09.610366 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 23157
I20260925 15:18:09.610662 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:44727
--local_ip_for_outbound_sockets=127.8.43.129
--tserver_master_addrs=127.8.43.190:44259
--webserver_port=44973
--webserver_interface=127.8.43.129
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:09.615090 23225 log.cc:824] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:09.617275 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:18:09.617573 23225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe: Bootstrap complete.
I20260925 15:18:09.619845 23225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:09.620086 23225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b935b5357f24cb69ab957fa3470c3fe, State: Initialized, Role: FOLLOWER
I20260925 15:18:09.620280 23225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:09.620373 23225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:09.620424 23225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:09.620472 23225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:18:09.621397 23225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:09.621520 23225 leader_election.cc:302] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b935b5357f24cb69ab957fa3470c3fe; no voters:
I20260925 15:18:09.621706 23225 leader_election.cc:288] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:18:09.621738 23229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:18:09.621840 23229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [term 3 LEADER]: Becoming Leader. State: Replica: 2b935b5357f24cb69ab957fa3470c3fe, State: Running, Role: LEADER
I20260925 15:18:09.621932 23229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } }
I20260925 15:18:09.621973 23225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:09.622174 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2b935b5357f24cb69ab957fa3470c3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } } }
I20260925 15:18:09.622257 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:09.622462 23231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b935b5357f24cb69ab957fa3470c3fe. Latest consensus state: current_term: 3 leader_uuid: "2b935b5357f24cb69ab957fa3470c3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b935b5357f24cb69ab957fa3470c3fe" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44259 } } }
I20260925 15:18:09.622663 23231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b935b5357f24cb69ab957fa3470c3fe [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:09.622773 23233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:09.623533 23233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:09.623632 23233 catalog_manager.cc:1259] Loaded cluster ID: 2dc7275813fb42288605e1684a8cfc2a
I20260925 15:18:09.623757 23233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:18:09.624228 23233 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f66223729ad google::LogMessage::Fail() at ??:0
@ 0x7f662237588a google::LogMessage::SendToLog() at ??:0
@ 0x7f66223724de google::LogMessage::Flush() at ??:0
@ 0x7f6622373a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f662a2930f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f662a29c8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f66242400d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6624235954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f66250996db start_thread at ??:0
@ 0x7f6620e7061f clone at ??:0
W20260925 15:18:09.713780 23227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:09.713964 23227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:09.713994 23227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:09.715104 23227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:09.715140 23227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:09.715166 23227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:18:09.716614 23227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:44727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=44973
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:09.716842 23227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:09.717096 23227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:09.722056 23262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.722175 23264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.722064 23261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:09.722440 23227 server_base.cc:1034] running on GCE node
I20260925 15:18:09.722680 23227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:09.723050 23227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:09.724241 23227 hybrid_clock.cc:648] HybridClock initialized: now 1790349489724228 us; error 27 us; skew 500 ppm
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:09.728193 23227 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:18:09.729208 23227 webserver.cc:460] Webserver started at http://127.8.43.129:44973/ using document root <none> and password file <none>
I20260925 15:18:09.729672 23227 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:09.729794 23227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:09.732012 23227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:18:09.732972 23271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.733198 23227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:09.733280 23227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "a8b8b2fff5f343b58c65f000d1e139c5"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "a45bcc6dece7b8c6f40b94a759a00a7f"
server_key_iv: "b6f718a61183bce168c6ffa0d3e54257"
server_key_version: "encryptionkey@0"
I20260925 15:18:09.733687 23227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:09.747767 23227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:09.750209 23227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:09.750332 23227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:09.750631 23227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:09.751143 23278 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:18:09.753224 23227 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:18:09.753302 23227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:18:09.753357 23227 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:18:09.754031 23227 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:18:09.754093 23227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:18:09.754163 23278 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Bootstrap starting.
I20260925 15:18:09.761965 23227 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:44727
I20260925 15:18:09.762094 23385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:44727 every 8 connection(s)
I20260925 15:18:09.762409 23227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:18:09.765444 23278 log.cc:824] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:09.767660 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 23227
I20260925 15:18:09.767917 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:34569
--local_ip_for_outbound_sockets=127.8.43.130
--tserver_master_addrs=127.8.43.190:44259
--webserver_port=46157
--webserver_interface=127.8.43.130
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:09.768347 23278 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:18:09.768687 23278 tablet_bootstrap.cc:492] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Bootstrap complete.
I20260925 15:18:09.769979 23278 ts_tablet_manager.cc:1397] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260925 15:18:09.771543 23278 raft_consensus.cc:348] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.771867 23278 raft_consensus.cc:729] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b8b2fff5f343b58c65f000d1e139c5, State: Initialized, Role: FOLLOWER
I20260925 15:18:09.772033 23278 consensus_queue.cc:260] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:09.772300 23278 ts_tablet_manager.cc:1428] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260925 15:18:09.850512 23390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:09.850776 23390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:09.850816 23390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:09.852372 23390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:09.852442 23390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:09.852484 23390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:18:09.853924 23390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:34569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=46157
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:09.854183 23390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:09.854399 23390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:09.855175 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:09.763234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:41663)
0925 15:18:09.763453 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.763511 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:09.763946 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:09.854963 (+ 91017us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
W20260925 15:18:09.855469 23386 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:44259 (0 consecutive failures): Network error: Failed to ping master at 127.8.43.190:44259: Client connection negotiation failed: client connection to 127.8.43.190:44259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:18:09.855860 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.855706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:56301)
0925 15:18:09.855773 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.855825 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:18:09.856253 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.856149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:39479)
0925 15:18:09.856199 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.856227 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 15:18:09.856369 23386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:18:09.859459 23399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.859493 23396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:09.859608 23390 server_base.cc:1034] running on GCE node
W20260925 15:18:09.859714 23397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:09.859899 23390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:09.860175 23390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:09.861348 23390 hybrid_clock.cc:648] HybridClock initialized: now 1790349489861327 us; error 32 us; skew 500 ppm
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:09.863750 23390 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:18:09.864198 23390 webserver.cc:460] Webserver started at http://127.8.43.130:46157/ using document root <none> and password file <none>
I20260925 15:18:09.864427 23390 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:09.864494 23390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:09.865761 23390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:09.866299 23406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.866463 23390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.866559 23390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "90bac09511cb4962b9f389494ef97f67"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "be47a0806bddab07394463fa0fbf26ac"
server_key_iv: "4e4e1545aedb31e4540859b6a5dec7d7"
server_key_version: "encryptionkey@0"
I20260925 15:18:09.866886 23390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:09.878127 23390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:09.881057 23390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:09.881203 23390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:09.881457 23390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:09.881812 23390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:09.881855 23390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.881898 23390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:09.881924 23390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.888674 23390 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:34569
I20260925 15:18:09.888728 23519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:34569 every 8 connection(s)
I20260925 15:18:09.889071 23390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:09.890096 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.889715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:52745)
0925 15:18:09.889861 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.889926 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260925 15:18:09.890496 23520 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:44259 (0 consecutive failures): Network error: Failed to ping master at 127.8.43.190:44259: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
I20260925 15:18:09.890776 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.890677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:59341)
0925 15:18:09.890725 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.890751 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 15:18:09.891042 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:09.890961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:43617)
0925 15:18:09.891003 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:09.891024 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:18:09.891144 23520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:18:09.893335 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 23390
I20260925 15:18:09.893560 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:41877
--local_ip_for_outbound_sockets=127.8.43.131
--tserver_master_addrs=127.8.43.190:44259
--webserver_port=45859
--webserver_interface=127.8.43.131
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:09.974879 23523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:09.975086 23523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:09.975109 23523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:09.976293 23523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:09.976338 23523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:09.976364 23523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:18:09.977815 23523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:41877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=45859
--tserver_master_addrs=127.8.43.190:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:09.978061 23523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:09.978345 23523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:09.983533 23531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.983537 23528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:09.983546 23529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:09.983570 23523 server_base.cc:1034] running on GCE node
I20260925 15:18:09.983989 23523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:09.984267 23523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:09.985428 23523 hybrid_clock.cc:648] HybridClock initialized: now 1790349489985427 us; error 38 us; skew 500 ppm
Sep 25 15:18:09 dist-test-slave-c5d6 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:09.987923 23523 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:18:09.988339 23523 webserver.cc:460] Webserver started at http://127.8.43.131:45859/ using document root <none> and password file <none>
I20260925 15:18:09.988517 23523 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:09.988556 23523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:09.989823 23523 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:09.990401 23538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:09.990653 23523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:18:09.990741 23523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "106c7c50ad1a4514bd424d7843ec2e88"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-c5d6"
server_key: "4a5d0bccca7797987281973866d20f00"
server_key_iv: "f685846fe48c6ef5e85e92e29ed24e13"
server_key_version: "encryptionkey@0"
I20260925 15:18:09.991065 23523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:10.002177 23523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:10.004400 23523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:10.004503 23523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:10.004673 23523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:10.004899 23523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:10.004928 23523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:10.004948 23523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:10.004961 23523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:10.010663 23523 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:41877
I20260925 15:18:10.010746 23651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:41877 every 8 connection(s)
I20260925 15:18:10.011133 23523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:18:10.012235 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:10.011846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:41291)
0925 15:18:10.012022 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:10.012093 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
W20260925 15:18:10.012645 23652 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:44259 (0 consecutive failures): Network error: Failed to ping master at 127.8.43.190:44259: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
I20260925 15:18:10.012924 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:10.012833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:46821)
0925 15:18:10.012876 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:10.012903 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 15:18:10.013206 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:10.013130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:49383)
0925 15:18:10.013169 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:10.013190 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:18:10.013326 23652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:18:10.018401 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 23523
I20260925 15:18:10.856974 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:10.856726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:47495)
0925 15:18:10.856871 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:10.856910 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":106,"threads_started":1}
I20260925 15:18:10.891633 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:10.891444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:58681)
0925 15:18:10.891552 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:10.891583 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260925 15:18:11.009483 23658 raft_consensus.cc:482] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:11.009650 23658 raft_consensus.cc:504] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:11.010025 23658 leader_election.cc:288] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
I20260925 15:18:11.013842 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:11.013619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:59775)
0925 15:18:11.013742 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:11.013772 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
Sep 25 15:18:11 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.130@KRBTEST.COM
Sep 25 15:18:11 dist-test-slave-c5d6 krb5kdc[22090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.131@KRBTEST.COM
I20260925 15:18:11.021560 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:11.010210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:41877 (local address 127.8.43.129:33159)
0925 15:18:11.010402 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:11.010412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:11.010502 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:11.011355 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:11.011356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:11.011773 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:11.012080 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:11.012087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:11.013792 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:11.013793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:11.014728 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:11.014731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:11.015384 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:11.015385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:11.015519 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:11.016107 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:11.016116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:11.018041 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:11.020005 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:11.020006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:11.020006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:11.020244 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:11.020531 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:11.020532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:11.020532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:11.020648 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:11.021035 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:11.021042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:11.021160 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 15:18:11.021266 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:18:11.021457 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":59,"thread_start_us":53,"threads_started":1}
I20260925 15:18:11.021574 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:11.010333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:60677 (local address 127.8.43.130:34569)
0925 15:18:11.010515 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:11.010528 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:11.010566 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:18:11.011014 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:11.011017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:11.011058 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:11.011212 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:11.012212 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:11.013673 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:11.014833 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:11.015300 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:11.015418 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:11.017814 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:11.017819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:11.017820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:11.017832 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:11.019794 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:11.020329 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:11.020333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:11.020333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:11.020426 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:11.020708 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:11.020709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:11.020709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:11.020953 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:11.021042 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:11.021322 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:18:11.021444 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260925 15:18:11.021584 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:11.010385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:33159 (local address 127.8.43.131:41877)
0925 15:18:11.010537 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:11.010549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:11.010571 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:18:11.011110 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:11.011113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:11.011242 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:11.011368 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:11.012211 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:11.013673 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:11.014838 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:11.015294 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:11.015406 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:11.018181 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:11.018184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:11.018184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:11.018194 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:11.019916 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:11.020342 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:11.020350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:11.020350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:11.020432 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:11.020729 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:11.020730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:11.020731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:11.020953 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:11.021039 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:11.021334 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 15:18:11.021471 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260925 15:18:11.021700 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:11.010203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:34569 (local address 127.8.43.129:60677)
0925 15:18:11.010245 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:11.010257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:11.010318 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:11.011259 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:11.011262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:11.011786 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:11.012080 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:11.012087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:11.013826 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:11.013827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:11.014730 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:11.014733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:11.015391 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:11.015392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:11.015519 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:11.016165 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:11.016170 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:11.017681 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:11.019883 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:11.019891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:11.019893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:11.020243 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:11.020529 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:11.020529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:11.020530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:11.020645 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:11.021041 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:11.021042 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:11.021160 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 15:18:11.021266 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:18:11.021540 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":159}
W20260925 15:18:11.023108 23273 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
W20260925 15:18:11.023249 23272 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
I20260925 15:18:11.023303 23272 leader_election.cc:302] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8b8b2fff5f343b58c65f000d1e139c5; no voters: 106c7c50ad1a4514bd424d7843ec2e88, 90bac09511cb4962b9f389494ef97f67
I20260925 15:18:11.023411 23658 raft_consensus.cc:2726] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:18:11.857686 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:11.857415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:35657)
0925 15:18:11.857595 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:11.857626 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":125,"threads_started":1}
I20260925 15:18:11.892246 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:11.892066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:46447)
0925 15:18:11.892176 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:11.892209 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260925 15:18:12.014528 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:12.014258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:58707)
0925 15:18:12.014440 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:12.014480 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260925 15:18:12.665686 23666 raft_consensus.cc:482] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:12.665812 23666 raft_consensus.cc:504] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:12.666030 23666 leader_election.cc:288] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
W20260925 15:18:12.666457 23272 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
W20260925 15:18:12.666520 23273 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
I20260925 15:18:12.666548 23272 leader_election.cc:302] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8b8b2fff5f343b58c65f000d1e139c5; no voters: 106c7c50ad1a4514bd424d7843ec2e88, 90bac09511cb4962b9f389494ef97f67
I20260925 15:18:12.666612 23666 raft_consensus.cc:2726] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:18:12.858371 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:12.858092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:33109)
0925 15:18:12.858265 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:12.858303 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260925 15:18:12.892875 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:12.892656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:42989)
0925 15:18:12.892805 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:12.892838 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260925 15:18:13.015298 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:13.015057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:47277)
0925 15:18:13.015185 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:13.015245 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260925 15:18:13.858994 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:13.858787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:44607)
0925 15:18:13.858906 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:13.858940 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260925 15:18:13.893579 23671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:13.893368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:57409)
0925 15:18:13.893501 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:13.893536 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260925 15:18:14.015982 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:14.015796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:43281)
0925 15:18:14.015908 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:14.015943 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260925 15:18:14.859706 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:14.859500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:46827)
0925 15:18:14.859631 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:14.859661 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260925 15:18:14.894197 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:14.894003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:35059)
0925 15:18:14.894123 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:14.894156 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260925 15:18:15.016598 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:15.016398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:53191)
0925 15:18:15.016514 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:15.016555 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260925 15:18:15.860378 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:15.860148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:38781)
0925 15:18:15.860285 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:15.860322 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260925 15:18:15.894797 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:15.894586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:49455)
0925 15:18:15.894704 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:15.894755 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260925 15:18:15.909852 23678 raft_consensus.cc:482] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:15.909956 23678 raft_consensus.cc:504] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:15.910391 23678 leader_election.cc:288] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
W20260925 15:18:15.910773 23273 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
W20260925 15:18:15.910848 23272 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
I20260925 15:18:15.910921 23272 leader_election.cc:302] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8b8b2fff5f343b58c65f000d1e139c5; no voters: 106c7c50ad1a4514bd424d7843ec2e88, 90bac09511cb4962b9f389494ef97f67
I20260925 15:18:15.911039 23678 raft_consensus.cc:2726] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:18:16.017333 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:16.017084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:48299)
0925 15:18:16.017247 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:16.017286 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260925 15:18:16.861128 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:16.860826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:58357)
0925 15:18:16.860982 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:16.861062 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260925 15:18:16.895470 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:16.895281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:42265)
0925 15:18:16.895397 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:16.895431 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260925 15:18:17.018152 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:17.017868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:45449)
0925 15:18:17.018064 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:17.018106 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":109,"threads_started":1}
I20260925 15:18:17.861864 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:17.861626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:34499)
0925 15:18:17.861782 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:17.861818 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260925 15:18:17.896106 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:17.895905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:45457)
0925 15:18:17.896026 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:17.896062 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260925 15:18:18.018913 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:18.018666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:35147)
0925 15:18:18.018812 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:18.018862 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260925 15:18:18.862515 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:18.862314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:33467)
0925 15:18:18.862431 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:18.862467 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260925 15:18:18.896818 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:18.896589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:58773)
0925 15:18:18.896736 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:18.896774 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260925 15:18:18.948884 23688 raft_consensus.cc:482] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:18.948993 23688 raft_consensus.cc:504] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:18.949167 23688 leader_election.cc:288] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
W20260925 15:18:18.949621 23272 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
W20260925 15:18:18.949703 23273 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
I20260925 15:18:18.949750 23273 leader_election.cc:302] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8b8b2fff5f343b58c65f000d1e139c5; no voters: 106c7c50ad1a4514bd424d7843ec2e88, 90bac09511cb4962b9f389494ef97f67
I20260925 15:18:18.949844 23688 raft_consensus.cc:2726] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:18:19.019718 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:19.019462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:60493)
0925 15:18:19.019622 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:19.019670 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260925 15:18:19.863271 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:19.862997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:44567)
0925 15:18:19.863143 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:19.863179 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260925 15:18:19.897454 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:19.897266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:41723)
0925 15:18:19.897386 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:19.897409 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260925 15:18:20.020509 23692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:20.020241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:48937)
0925 15:18:20.020420 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:20.020461 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":114,"threads_started":1}
I20260925 15:18:20.863936 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:20.863715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:55479)
0925 15:18:20.863851 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:20.863889 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260925 15:18:20.898087 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:20.897869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:59711)
0925 15:18:20.898015 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:20.898045 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260925 15:18:21.021221 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:21.020977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:46061)
0925 15:18:21.021118 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:21.021160 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260925 15:18:21.864674 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:21.864406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:42979)
0925 15:18:21.864575 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:21.864619 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260925 15:18:21.898792 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:21.898578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:54037)
0925 15:18:21.898706 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:21.898746 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260925 15:18:22.021940 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:22.021719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:47269)
0925 15:18:22.021858 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:22.021892 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260925 15:18:22.865386 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:22.865169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:40715)
0925 15:18:22.865305 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:22.865337 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260925 15:18:22.899502 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:22.899301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:33243)
0925 15:18:22.899418 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:22.899458 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260925 15:18:23.022624 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:23.022387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:44539)
0925 15:18:23.022530 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:23.022568 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260925 15:18:23.866070 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:23.865838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:51213)
0925 15:18:23.865981 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:23.866017 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260925 15:18:23.900179 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:23.899978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:46425)
0925 15:18:23.900088 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:23.900126 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260925 15:18:24.023414 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:24.023147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:55173)
0925 15:18:24.023324 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:24.023365 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":114,"threads_started":1}
I20260925 15:18:24.518846 23705 raft_consensus.cc:482] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:24.518944 23705 raft_consensus.cc:504] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106c7c50ad1a4514bd424d7843ec2e88" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 41877 } } peers { permanent_uuid: "90bac09511cb4962b9f389494ef97f67" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 34569 } } peers { permanent_uuid: "a8b8b2fff5f343b58c65f000d1e139c5" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44727 } }
I20260925 15:18:24.519160 23705 leader_election.cc:288] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877), 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569)
W20260925 15:18:24.519567 23272 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90bac09511cb4962b9f389494ef97f67 (127.8.43.130:34569): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
W20260925 15:18:24.519634 23273 leader_election.cc:341] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 106c7c50ad1a4514bd424d7843ec2e88 (127.8.43.131:41877): Not found: Tablet not found: 036b80766fa941d1ab86ea5be89968d2
I20260925 15:18:24.519665 23273 leader_election.cc:302] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8b8b2fff5f343b58c65f000d1e139c5; no voters: 106c7c50ad1a4514bd424d7843ec2e88, 90bac09511cb4962b9f389494ef97f67
I20260925 15:18:24.519749 23705 raft_consensus.cc:2726] T 036b80766fa941d1ab86ea5be89968d2 P a8b8b2fff5f343b58c65f000d1e139c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:18:24.866796 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:24.866592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.129:35619)
0925 15:18:24.866718 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:24.866751 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260925 15:18:24.900929 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:24.900729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.130:38593)
0925 15:18:24.900854 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:24.900887 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 15:18:25.024272 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:25.024044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44259 (local address 127.8.43.131:60775)
0925 15:18:25.024173 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:25.024221 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260925 15:18:25.028728 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 23227
I20260925 15:18:25.033478 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 23390
I20260925 15:18:25.037580 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 23523
2026-09-25T15:18:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18248 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:25 dist-test-slave-c5d6 krb5kdc[23711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:25 dist-test-slave-c5d6 krb5kdc[23711](info): set up 2 sockets
Sep 25 15:18:25 dist-test-slave-c5d6 krb5kdc[23711](info): commencing operation
krb5kdc: starting...
W20260925 15:18:26.087270 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:18:26.204089 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44753
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:26.285106 23724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:26.285315 23724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:26.285337 23724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:26.286509 23724 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:18:26.286549 23724 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:18:26.286564 23724 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:18:26.286577 23724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:26.286618 23724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:26.286650 23724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:26.286667 23724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:26.286689 23724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:26.288240 23724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:26.288516 23724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:26.288746 23724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:26.293804 23730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:26.293831 23724 server_base.cc:1034] running on GCE node
W20260925 15:18:26.293857 23732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:26.293835 23729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:26.294179 23724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:26.294462 23724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:26.295621 23724 hybrid_clock.cc:648] HybridClock initialized: now 1790349506295602 us; error 41 us; skew 500 ppm
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:26.297962 23724 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:18:26.298385 23724 webserver.cc:460] Webserver started at http://127.8.43.190:32961/ using document root <none> and password file <none>
I20260925 15:18:26.298621 23724 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:26.298688 23724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:26.298837 23724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:26.299957 23724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "c99dcc23f0dc47848037ff660f7ed64d"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "fb7357d990e6928909ab93f89a2205fc"
server_key_iv: "90baa3c04afc2db869e2021a61ead91c"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.300312 23724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "c99dcc23f0dc47848037ff660f7ed64d"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "fb7357d990e6928909ab93f89a2205fc"
server_key_iv: "90baa3c04afc2db869e2021a61ead91c"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.301581 23724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:18:26.302292 23739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.302456 23724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:18:26.302557 23724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "c99dcc23f0dc47848037ff660f7ed64d"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "fb7357d990e6928909ab93f89a2205fc"
server_key_iv: "90baa3c04afc2db869e2021a61ead91c"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.302631 23724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:26.317703 23724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:26.319777 23724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:26.319937 23724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:26.323859 23724 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44753
I20260925 15:18:26.323910 23791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44753 every 8 connection(s)
I20260925 15:18:26.324270 23724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:26.325142 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:26.327389 23792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Bootstrap starting.
I20260925 15:18:26.327962 23792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:26.328271 23792 log.cc:824] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:26.328924 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 23724
I20260925 15:18:26.329020 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:18:26.329022 23792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: No bootstrap required, opened a new log
I20260925 15:18:26.329252 8366 external_mini_cluster.cc:1428] Setting key d1597df3baccb8a32381b9d2b0082fd6
I20260925 15:18:26.331476 23792 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:18:26.331645 23792 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:26.331693 23792 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99dcc23f0dc47848037ff660f7ed64d, State: Initialized, Role: FOLLOWER
I20260925 15:18:26.331830 23792 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:18:26.331928 23792 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:26.331970 23792 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:26.332017 23792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:26.332521 23792 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:18:26.332644 23792 leader_election.cc:302] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99dcc23f0dc47848037ff660f7ed64d; no voters:
I20260925 15:18:26.332821 23792 leader_election.cc:288] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:26.332873 23797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:26.333052 23797 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 1 LEADER]: Becoming Leader. State: Replica: c99dcc23f0dc47848037ff660f7ed64d, State: Running, Role: LEADER
I20260925 15:18:26.333063 23792 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:26.333189 23797 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:18:26.333628 23798 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99dcc23f0dc47848037ff660f7ed64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } } }
I20260925 15:18:26.333619 23799 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99dcc23f0dc47848037ff660f7ed64d. Latest consensus state: current_term: 1 leader_uuid: "c99dcc23f0dc47848037ff660f7ed64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } } }
I20260925 15:18:26.333717 23798 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:26.333721 23799 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:26.334467 23804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:26.334921 23804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:26.337316 23804 catalog_manager.cc:1347] Generated new cluster ID: b1026819a7b44632b2f9eb4d80409e06
I20260925 15:18:26.337365 23804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:26.341296 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.329766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.8.43.190:44753)
0925 15:18:26.329941 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.329949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.329972 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:18:26.330397 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.330404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.330447 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.330554 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:26.331341 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.332684 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.333777 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.334446 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.334539 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.337549 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:26.337553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:26.337554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:26.337564 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:26.339644 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.340060 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.340063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.340063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.340164 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.340383 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.340384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.340384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.340669 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:26.340785 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.340982 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.341109 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260925 15:18:26.355000 23804 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:26.355698 23804 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:26.366190 23804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Generated new TSK 0
I20260925 15:18:26.366370 23804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:18:26.394337 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44753
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:26.475756 23818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:26.476008 23818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:26.476050 23818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:26.477209 23818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:26.477277 23818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:26.477324 23818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:18:26.478826 23818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:26.479099 23818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:26.479379 23818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:26.484503 23824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:26.484568 23823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:26.484724 23826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:26.484727 23818 server_base.cc:1034] running on GCE node
I20260925 15:18:26.484961 23818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:26.485261 23818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:26.486433 23818 hybrid_clock.cc:648] HybridClock initialized: now 1790349506486421 us; error 29 us; skew 500 ppm
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:26.489130 23818 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:18:26.489629 23818 webserver.cc:460] Webserver started at http://127.8.43.129:45731/ using document root <none> and password file <none>
I20260925 15:18:26.489871 23818 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:26.489938 23818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:26.490070 23818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:26.491107 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "2d67a3639df3470a80d14e735ca63a2f"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "74a6f90d8b0056ea4dac61d4fe9a4f14"
server_key_iv: "5ec3cff8ae687f26ca699b05311e3192"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.491489 23818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "2d67a3639df3470a80d14e735ca63a2f"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "74a6f90d8b0056ea4dac61d4fe9a4f14"
server_key_iv: "5ec3cff8ae687f26ca699b05311e3192"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.492794 23818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:26.493531 23833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.493731 23818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:26.493803 23818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "2d67a3639df3470a80d14e735ca63a2f"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "74a6f90d8b0056ea4dac61d4fe9a4f14"
server_key_iv: "5ec3cff8ae687f26ca699b05311e3192"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.493894 23818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:26.514914 23818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:26.516994 23818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:26.517149 23818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:26.517376 23818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:26.517696 23818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:26.517750 23818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.517792 23818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:26.517819 23818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.523842 23818 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:40599
I20260925 15:18:26.523888 23946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:40599 every 8 connection(s)
I20260925 15:18:26.524199 23818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:18:26.529386 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 23818
I20260925 15:18:26.529497 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:18:26.529733 8366 external_mini_cluster.cc:1428] Setting key 5e8cd327a12a7cc067864bfed4b0653e
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:26.533691 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.524936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:60705 (local address 127.8.43.190:44753)
0925 15:18:26.525000 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.525002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.525195 (+ 193us) server_negotiation.cc:407] Connection header received
0925 15:18:26.525682 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.525683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.525693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.525761 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:26.526667 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.527181 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.528204 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.528401 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.528459 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.530739 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:26.530740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:26.530741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:26.530748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:26.532247 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.532678 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.532680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.532680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.532739 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.532991 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.532992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.532993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.533105 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:26.533183 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.533509 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.533603 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:18:26.533879 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.524911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:60705)
0925 15:18:26.525062 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.525113 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:26.525590 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:26.525816 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:26.525820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:26.526381 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:26.526572 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.526576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.527327 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.527327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.528125 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.528127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.528476 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.528477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.528570 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.529006 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:26.529014 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:26.530636 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:26.532348 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:26.532355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:26.532357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:26.532586 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:26.532806 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:26.532807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:26.532808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:26.532921 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:26.533175 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:26.533182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:26.533321 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 15:18:26.533487 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 15:18:26.533641 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260925 15:18:26.534574 23947 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44753
I20260925 15:18:26.534685 23947 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:26.534878 23947 heartbeater.cc:511] Master 127.8.43.190:44753 requested a full tablet report, sending...
I20260925 15:18:26.535488 23756 ts_manager.cc:194] Registered new tserver with Master: 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599)
I20260925 15:18:26.536445 23756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:60705
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:18:26.553910 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44753
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:26.633348 23952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:26.633562 23952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:26.633602 23952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:26.634954 23952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:26.635010 23952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:26.635053 23952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:18:26.636538 23952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:26.636783 23952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:26.637012 23952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:26.641929 23957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:26.641925 23958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:26.641933 23960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:26.642072 23952 server_base.cc:1034] running on GCE node
I20260925 15:18:26.642352 23952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:26.642553 23952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:26.643715 23952 hybrid_clock.cc:648] HybridClock initialized: now 1790349506643693 us; error 35 us; skew 500 ppm
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:26.646199 23952 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:18:26.646643 23952 webserver.cc:460] Webserver started at http://127.8.43.130:34149/ using document root <none> and password file <none>
I20260925 15:18:26.646867 23952 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:26.646910 23952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:26.647068 23952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:26.648227 23952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "ebcec57e0f8644a5a5e2ad7607dffdce"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "adc75735aa934988b220c6096757f989"
server_key_iv: "16b491cf2a824bd5766f0692918a5d1b"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.648582 23952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "ebcec57e0f8644a5a5e2ad7607dffdce"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "adc75735aa934988b220c6096757f989"
server_key_iv: "16b491cf2a824bd5766f0692918a5d1b"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.649875 23952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:18:26.650563 23967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.650750 23952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:18:26.650844 23952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "ebcec57e0f8644a5a5e2ad7607dffdce"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "adc75735aa934988b220c6096757f989"
server_key_iv: "16b491cf2a824bd5766f0692918a5d1b"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.650921 23952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:26.668985 23952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:26.670823 23952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:26.670976 23952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:26.671198 23952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:26.671535 23952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:26.671595 23952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.671636 23952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:26.671666 23952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.677366 23952 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:32993
I20260925 15:18:26.677412 24080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:32993 every 8 connection(s)
I20260925 15:18:26.677721 23952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:26.677911 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 23952
I20260925 15:18:26.678010 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:18:26.678215 8366 external_mini_cluster.cc:1428] Setting key 87ed7d1f80b963a2980aec234d7dd3a3
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
I20260925 15:18:26.688540 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.678478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:35501 (local address 127.8.43.190:44753)
0925 15:18:26.678529 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.678531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.678982 (+ 451us) server_negotiation.cc:407] Connection header received
0925 15:18:26.679542 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.679544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.679556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.679622 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:26.680573 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.681424 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.683020 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.683204 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.683307 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.685665 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:26.685667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:26.685667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:26.685675 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:26.687125 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.687566 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.687568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.687568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.687629 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.687912 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.687913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.687913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.688026 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:26.688102 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.688371 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.688449 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:18:26.688741 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.678466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:35501)
0925 15:18:26.678830 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.678894 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:26.679451 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:26.679686 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:26.679690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:26.680288 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:26.680472 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.680476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.681548 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.681549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.682941 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.682944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.683333 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.683334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.683486 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.683973 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:26.683984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:26.685558 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:26.687249 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:26.687254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:26.687257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:26.687472 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:26.687704 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:26.687705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:26.687705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:26.687835 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:26.688093 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:26.688099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:26.688239 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 15:18:26.688348 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:18:26.688503 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":85,"threads_started":1}
I20260925 15:18:26.689284 24081 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
I20260925 15:18:26.689390 24081 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:26.689589 24081 heartbeater.cc:511] Master 127.8.43.190:44753 requested a full tablet report, sending...
I20260925 15:18:26.689990 23756 ts_manager.cc:194] Registered new tserver with Master: ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993)
I20260925 15:18:26.690390 23756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:35501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:18:26.705674 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44753
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:26.786525 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:26.786720 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:26.786741 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:26.787925 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:26.787967 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:26.787993 24086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:18:26.789453 24086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:26.789676 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:26.789932 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:26.794663 24092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:26.794677 24091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:26.794695 24094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:26.794868 24086 server_base.cc:1034] running on GCE node
I20260925 15:18:26.795106 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:26.795396 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:26.796537 24086 hybrid_clock.cc:648] HybridClock initialized: now 1790349506796524 us; error 28 us; skew 500 ppm
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:26.799324 24086 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:18:26.799768 24086 webserver.cc:460] Webserver started at http://127.8.43.131:46081/ using document root <none> and password file <none>
I20260925 15:18:26.799994 24086 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:26.800040 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:26.800129 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:26.801064 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "d87a1123f3384529bbacdc62ce577705"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "5c8de14c2f4947d792ee89d655e41ed2"
server_key_iv: "648adf8110631e43c5bbab518de9195e"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.801352 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "d87a1123f3384529bbacdc62ce577705"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "5c8de14c2f4947d792ee89d655e41ed2"
server_key_iv: "648adf8110631e43c5bbab518de9195e"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.802610 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:18:26.803318 24101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.803491 24086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:26.803597 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "d87a1123f3384529bbacdc62ce577705"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "5c8de14c2f4947d792ee89d655e41ed2"
server_key_iv: "648adf8110631e43c5bbab518de9195e"
server_key_version: "encryptionkey@0"
I20260925 15:18:26.803670 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:26.811906 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:26.813642 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:26.813794 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:26.813992 24086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:26.814314 24086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:26.814348 24086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.814369 24086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:26.814425 24086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:26.820205 24086 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:39127
I20260925 15:18:26.820243 24214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:39127 every 8 connection(s)
I20260925 15:18:26.820583 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:18:26 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349506, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:26.829346 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.821265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:35127 (local address 127.8.43.190:44753)
0925 15:18:26.821322 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.821325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.821559 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:18:26.822138 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.822139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.822152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.822201 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:26.823127 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.823713 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.824697 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.824916 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.824988 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.827031 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:26.827032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:26.827033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:26.827039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:26.828214 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.828558 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.828559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.828559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.828623 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.828838 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.828839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.828839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.828946 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:26.828999 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.829225 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.829284 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:18:26.829531 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.821241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:35127)
0925 15:18:26.821412 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.821467 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:26.822004 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:26.822263 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:26.822267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:26.822849 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:26.823029 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.823033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.823803 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.823804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.824577 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.824579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.824994 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.824995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.825117 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.825607 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:26.825615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:26.826944 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:26.828281 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:26.828287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:26.828288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:26.828493 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:26.828681 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:26.828681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:26.828682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:26.828774 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:26.829001 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:26.829005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:26.829120 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 15:18:26.829198 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:18:26.829307 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260925 15:18:26.829974 24215 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44753
I20260925 15:18:26.830051 24215 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:26.830241 24215 heartbeater.cc:511] Master 127.8.43.190:44753 requested a full tablet report, sending...
I20260925 15:18:26.830420 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 24086
I20260925 15:18:26.830529 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:18:26.830667 23756 ts_manager.cc:194] Registered new tserver with Master: d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127)
I20260925 15:18:26.830730 8366 external_mini_cluster.cc:1428] Setting key 76a7cb6605636dfdb8c4a3fc7fce34f8
I20260925 15:18:26.831051 23756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:35127
I20260925 15:18:26.831713 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:18:26.839115 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.833070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.8.43.190:44753)
0925 15:18:26.833107 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.833109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.833243 (+ 134us) server_negotiation.cc:407] Connection header received
0925 15:18:26.833315 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.833316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.833325 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.833372 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:26.834035 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.834560 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.835424 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.835633 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.835713 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.836856 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:26.836857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:26.836857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:26.836864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:26.837958 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.838329 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.838330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.838331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.838387 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:26.838589 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:26.838590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:26.838590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:26.838724 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:26.838832 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.838946 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.839020 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:18:26.840512 23756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:26.842064 23756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:26.848526 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.844482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.8.43.131:39127)
0925 15:18:26.844702 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.844705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.844782 (+ 77us) server_negotiation.cc:407] Connection header received
0925 15:18:26.844867 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.844869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.844886 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.844973 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:26.845246 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.846000 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.847383 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.848058 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.848160 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.848222 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.848398 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.848437 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":130,"threads_started":1}
I20260925 15:18:26.848526 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.844265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:39127 (local address 127.0.0.1:51230)
0925 15:18:26.844698 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.844710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:26.844784 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:26.845063 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:26.845066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:26.845069 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:26.845154 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.845155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.846112 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.846113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.847297 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.847300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.848163 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.848164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.848259 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.848273 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:18:26.848381 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:18:26.848437 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":54,"threads_started":1}
I20260925 15:18:26.848927 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.844285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:40599 (local address 127.0.0.1:50428)
0925 15:18:26.844872 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.844880 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:26.844922 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:26.845244 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:26.845245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:26.845248 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:26.845294 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.845296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.846596 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.846597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.847754 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.847757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.848692 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.848693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.848754 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.848758 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:18:26.848812 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:18:26.848851 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":36,"threads_started":1}
I20260925 15:18:26.849030 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.844262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:32993 (local address 127.0.0.1:57120)
0925 15:18:26.844667 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.844709 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:26.844784 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:26.845156 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:26.845157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:26.845159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:26.845206 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.845208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.846569 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.846570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.847626 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.847628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.848604 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.848774 (+ 170us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.848833 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.848837 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:18:26.848883 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:18:26.848912 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260925 15:18:26.849136 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.844440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.8.43.130:32993)
0925 15:18:26.844632 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.844637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.844905 (+ 268us) server_negotiation.cc:407] Connection header received
0925 15:18:26.844956 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.844958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.844972 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.845059 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:26.845289 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.846446 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.847713 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.848440 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.848511 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.848567 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.848977 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.849026 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260925 15:18:26.849385 24149 tablet_service.cc:1467] Processing CreateTablet for tablet 03151cc11bc745869ac30a2f69c8d8c2 (DEFAULT_TABLE table=test-table [id=7587553968664186b823a880ca665e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:26.849748 24149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03151cc11bc745869ac30a2f69c8d8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:26.849758 24015 tablet_service.cc:1467] Processing CreateTablet for tablet 03151cc11bc745869ac30a2f69c8d8c2 (DEFAULT_TABLE table=test-table [id=7587553968664186b823a880ca665e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:26.850059 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03151cc11bc745869ac30a2f69c8d8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:26.851819 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.844495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.8.43.129:40599)
0925 15:18:26.844662 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.844667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.844981 (+ 314us) server_negotiation.cc:407] Connection header received
0925 15:18:26.845027 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.845029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.845045 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.845141 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:26.845384 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.846473 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.847852 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.848541 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.848613 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.848665 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.848834 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.848883 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260925 15:18:26.852411 23881 tablet_service.cc:1467] Processing CreateTablet for tablet 03151cc11bc745869ac30a2f69c8d8c2 (DEFAULT_TABLE table=test-table [id=7587553968664186b823a880ca665e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:26.852720 23881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03151cc11bc745869ac30a2f69c8d8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:26.853322 24230 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: Bootstrap starting.
I20260925 15:18:26.853549 24229 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Bootstrap starting.
I20260925 15:18:26.854250 24229 tablet_bootstrap.cc:654] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:26.854854 24229 log.cc:824] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:26.855674 24229 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: No bootstrap required, opened a new log
I20260925 15:18:26.855681 24230 tablet_bootstrap.cc:654] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:26.855756 24229 ts_tablet_manager.cc:1397] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:26.856312 24230 log.cc:824] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:26.856587 24232 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Bootstrap starting.
I20260925 15:18:26.856978 24230 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: No bootstrap required, opened a new log
I20260925 15:18:26.857031 24230 ts_tablet_manager.cc:1397] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:18:26.857360 24232 tablet_bootstrap.cc:654] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:26.857834 24229 raft_consensus.cc:348] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.858048 24229 raft_consensus.cc:374] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:26.858095 24229 raft_consensus.cc:729] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcec57e0f8644a5a5e2ad7607dffdce, State: Initialized, Role: FOLLOWER
I20260925 15:18:26.858004 24232 log.cc:824] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:26.858220 24229 consensus_queue.cc:260] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.858544 24229 ts_tablet_manager.cc:1428] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:26.858402 24230 raft_consensus.cc:348] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.858577 24081 heartbeater.cc:503] Master 127.8.43.190:44753 was elected leader, sending a full tablet report...
I20260925 15:18:26.858600 24230 raft_consensus.cc:374] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:26.858646 24230 raft_consensus.cc:729] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87a1123f3384529bbacdc62ce577705, State: Initialized, Role: FOLLOWER
I20260925 15:18:26.858765 24230 consensus_queue.cc:260] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.859092 24230 ts_tablet_manager.cc:1428] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:26.859211 24215 heartbeater.cc:503] Master 127.8.43.190:44753 was elected leader, sending a full tablet report...
I20260925 15:18:26.859354 24232 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: No bootstrap required, opened a new log
I20260925 15:18:26.859467 24232 ts_tablet_manager.cc:1397] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:26.861142 24232 raft_consensus.cc:348] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.861280 24232 raft_consensus.cc:374] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:26.861305 24232 raft_consensus.cc:729] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d67a3639df3470a80d14e735ca63a2f, State: Initialized, Role: FOLLOWER
I20260925 15:18:26.861440 24232 consensus_queue.cc:260] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.861730 24232 ts_tablet_manager.cc:1428] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:26.861794 23947 heartbeater.cc:503] Master 127.8.43.190:44753 was elected leader, sending a full tablet report...
I20260925 15:18:26.900506 24234 raft_consensus.cc:482] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:26.900689 24234 raft_consensus.cc:504] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.901072 24234 leader_election.cc:288] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127), 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599)
I20260925 15:18:26.904132 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.901344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:35569 (local address 127.8.43.131:39127)
0925 15:18:26.901394 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.901396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.901570 (+ 174us) server_negotiation.cc:407] Connection header received
0925 15:18:26.901668 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.901670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.901678 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.901754 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:26.901902 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.902367 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.903372 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.903767 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.903860 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.903869 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.904027 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.904060 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:18:26.904142 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.901282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:39127 (local address 127.8.43.130:35569)
0925 15:18:26.901499 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.901507 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:26.901594 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:26.901755 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:26.901756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:26.901757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:26.901827 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.901831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.902448 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.902449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.903275 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.903278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.903861 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.903862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.903930 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.903933 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:18:26.904010 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:18:26.904060 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":14,"thread_start_us":125,"threads_started":1}
I20260925 15:18:26.904245 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.901282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:40599 (local address 127.8.43.130:41731)
0925 15:18:26.901346 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.901360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:26.901406 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:26.901667 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:26.901668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:26.901669 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:26.901751 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.901753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.902408 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.902409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.903283 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:26.903286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.903946 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.903947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:26.904056 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.904059 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:18:26.904135 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:18:26.904172 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":48}
I20260925 15:18:26.904245 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.901445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:41731 (local address 127.8.43.129:40599)
0925 15:18:26.901529 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.901531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.901540 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:18:26.901565 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.901566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.901578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.901661 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:26.901846 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.902322 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.903431 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.903845 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.903946 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.903955 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.904143 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.904180 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:18:26.904582 23901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03151cc11bc745869ac30a2f69c8d8c2" candidate_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d67a3639df3470a80d14e735ca63a2f" is_pre_election: true
I20260925 15:18:26.904606 24169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03151cc11bc745869ac30a2f69c8d8c2" candidate_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87a1123f3384529bbacdc62ce577705" is_pre_election: true
I20260925 15:18:26.904762 24169 raft_consensus.cc:2454] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebcec57e0f8644a5a5e2ad7607dffdce in term 0.
I20260925 15:18:26.904770 23901 raft_consensus.cc:2454] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebcec57e0f8644a5a5e2ad7607dffdce in term 0.
I20260925 15:18:26.905032 23969 leader_election.cc:302] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d67a3639df3470a80d14e735ca63a2f, ebcec57e0f8644a5a5e2ad7607dffdce; no voters:
I20260925 15:18:26.905259 24234 raft_consensus.cc:2781] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:26.905323 24234 raft_consensus.cc:482] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:26.905359 24234 raft_consensus.cc:3037] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:26.906246 24234 raft_consensus.cc:504] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.906424 24234 leader_election.cc:288] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [CANDIDATE]: Term 1 election: Requested vote from peers d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127), 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599)
I20260925 15:18:26.906558 24169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03151cc11bc745869ac30a2f69c8d8c2" candidate_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87a1123f3384529bbacdc62ce577705"
I20260925 15:18:26.906641 24169 raft_consensus.cc:3037] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:26.907361 23901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03151cc11bc745869ac30a2f69c8d8c2" candidate_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d67a3639df3470a80d14e735ca63a2f"
I20260925 15:18:26.907476 24169 raft_consensus.cc:2454] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebcec57e0f8644a5a5e2ad7607dffdce in term 1.
I20260925 15:18:26.907451 23901 raft_consensus.cc:3037] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:26.907657 23971 leader_election.cc:302] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d87a1123f3384529bbacdc62ce577705, ebcec57e0f8644a5a5e2ad7607dffdce; no voters:
I20260925 15:18:26.907753 24234 raft_consensus.cc:2781] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:26.907922 24234 raft_consensus.cc:686] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 LEADER]: Becoming Leader. State: Replica: ebcec57e0f8644a5a5e2ad7607dffdce, State: Running, Role: LEADER
I20260925 15:18:26.908025 24234 consensus_queue.cc:237] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:26.908532 23901 raft_consensus.cc:2454] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebcec57e0f8644a5a5e2ad7607dffdce in term 1.
I20260925 15:18:26.908918 23755 catalog_manager.cc:5495] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce reported cstate change: term changed from 0 to 1, leader changed from <none> to ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130). New cstate: current_term: 1 leader_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } health_report { overall_health: UNKNOWN } } }
I20260925 15:18:26.924674 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.922305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.8.43.130:32993)
0925 15:18:26.922364 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:26.922366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:26.922373 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:18:26.922411 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:26.922411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:26.922431 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:26.922519 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:26.922667 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.923136 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.923896 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:26.924106 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:26.924187 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:26.924348 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:26.924449 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:26.924539 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:26.924558 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 15:18:26.924576 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:18:26.926304 23901 raft_consensus.cc:1261] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Refusing update from remote peer ebcec57e0f8644a5a5e2ad7607dffdce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:26.926304 24169 raft_consensus.cc:1261] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 1 FOLLOWER]: Refusing update from remote peer ebcec57e0f8644a5a5e2ad7607dffdce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:26.926630 24234 consensus_queue.cc:1035] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:26.926769 24239 consensus_queue.cc:1035] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:18:26.928503 24082 tablet.cc:2369] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:18:26.929522 24242 mvcc.cc:205] Tried to move back new op lower bound from 7333271580367560704 to 7333271580295892992. Current Snapshot: MvccSnapshot[applied={T|T < 7333271580367560704}]
I20260925 15:18:26.931977 23755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:26.932080 23755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:26.932797 23755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Sending DeleteTablet for 3 replicas of tablet 03151cc11bc745869ac30a2f69c8d8c2
I20260925 15:18:26.933141 24015 tablet_service.cc:1514] Processing DeleteTablet for tablet 03151cc11bc745869ac30a2f69c8d8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:26 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:57120
I20260925 15:18:26.933200 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 23818
I20260925 15:18:26.933248 24149 tablet_service.cc:1514] Processing DeleteTablet for tablet 03151cc11bc745869ac30a2f69c8d8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:26 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:51230
I20260925 15:18:26.934054 24252 tablet_replica.cc:330] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: stopping tablet replica
I20260925 15:18:26.934170 24252 raft_consensus.cc:2229] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:26.934279 24252 raft_consensus.cc:2258] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:26.934739 24252 ts_tablet_manager.cc:1905] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:26.936749 24252 ts_tablet_manager.cc:1918] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:26.936837 24252 log.cc:1196] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/03151cc11bc745869ac30a2f69c8d8c2
I20260925 15:18:26.937108 24252 ts_tablet_manager.cc:1939] T 03151cc11bc745869ac30a2f69c8d8c2 P d87a1123f3384529bbacdc62ce577705: Deleting consensus metadata
I20260925 15:18:26.937501 23743 catalog_manager.cc:4841] TS d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127): tablet 03151cc11bc745869ac30a2f69c8d8c2 (table test-table [id=7587553968664186b823a880ca665e2d]) successfully deleted
I20260925 15:18:26.939345 24251 tablet_replica.cc:330] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: stopping tablet replica
I20260925 15:18:26.939455 24251 raft_consensus.cc:2229] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:18:26.939580 24251 raft_consensus.cc:2258] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:26.940016 24251 ts_tablet_manager.cc:1905] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:18:26.940024 23741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:40599 (error 108)
I20260925 15:18:26.940480 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.940368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:40599 (local address 127.0.0.1:50440)
0925 15:18:26.940419 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.940450 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:18:26.940539 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 23952
W20260925 15:18:26.943666 23741 catalog_manager.cc:4584] TS 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03151cc11bc745869ac30a2f69c8d8c2: Network error: Client connection negotiation failed: client connection to 127.8.43.129:40599: connect: Connection refused (error 111)
I20260925 15:18:26.945199 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:26.945090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:32993 (local address 127.0.0.1:57144)
0925 15:18:26.945145 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:26.945177 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.130:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:18:26.945273 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 24086
I20260925 15:18:26.950423 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 23724
I20260925 15:18:46.955446 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44753
--webserver_interface=127.8.43.190
--webserver_port=32961
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:47.037621 24254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:47.037855 24254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:47.037878 24254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:47.039047 24254 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:18:47.039088 24254 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:18:47.039103 24254 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:18:47.039113 24254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:47.039168 24254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:47.039186 24254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:47.039235 24254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:47.039258 24254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:47.040727 24254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=32961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:47.040982 24254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:47.041216 24254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:47.049774 24261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:47.049829 24263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:47.049921 24254 server_base.cc:1034] running on GCE node
W20260925 15:18:47.049862 24260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:47.050307 24254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:47.050625 24254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:47.051765 24254 hybrid_clock.cc:648] HybridClock initialized: now 1790349527051742 us; error 43 us; skew 500 ppm
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:47.054306 24254 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:18:47.054754 24254 webserver.cc:460] Webserver started at http://127.8.43.190:32961/ using document root <none> and password file <none>
I20260925 15:18:47.054996 24254 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:47.055053 24254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:47.056517 24254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:18:47.057135 24270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:47.057319 24254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:18:47.057420 24254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "c99dcc23f0dc47848037ff660f7ed64d"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "fb7357d990e6928909ab93f89a2205fc"
server_key_iv: "90baa3c04afc2db869e2021a61ead91c"
server_key_version: "encryptionkey@0"
I20260925 15:18:47.057806 24254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:47.075592 24254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:47.077984 24254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:47.078142 24254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:47.082206 24254 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44753
I20260925 15:18:47.082259 24322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44753 every 8 connection(s)
I20260925 15:18:47.082633 24254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:47.084167 24323 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:18:47.084844 24323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Bootstrap starting.
I20260925 15:18:47.090588 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 24254
I20260925 15:18:47.090875 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:40599
--local_ip_for_outbound_sockets=127.8.43.129
--tserver_master_addrs=127.8.43.190:44753
--webserver_port=45731
--webserver_interface=127.8.43.129
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:47.093024 24323 log.cc:824] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:47.095022 24323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:18:47.095373 24323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Bootstrap complete.
I20260925 15:18:47.097569 24323 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:18:47.097815 24323 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99dcc23f0dc47848037ff660f7ed64d, State: Initialized, Role: FOLLOWER
I20260925 15:18:47.097994 24323 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:18:47.098073 24323 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:47.098127 24323 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:47.098176 24323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:18:47.099157 24323 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:18:47.099288 24323 leader_election.cc:302] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99dcc23f0dc47848037ff660f7ed64d; no voters:
I20260925 15:18:47.099490 24323 leader_election.cc:288] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:18:47.099524 24327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:18:47.099640 24327 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 2 LEADER]: Becoming Leader. State: Replica: c99dcc23f0dc47848037ff660f7ed64d, State: Running, Role: LEADER
I20260925 15:18:47.099762 24323 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:47.099795 24327 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:18:47.100066 24327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99dcc23f0dc47848037ff660f7ed64d. Latest consensus state: current_term: 2 leader_uuid: "c99dcc23f0dc47848037ff660f7ed64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } } }
I20260925 15:18:47.100142 24327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:47.100075 24329 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c99dcc23f0dc47848037ff660f7ed64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } } }
I20260925 15:18:47.100347 24332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:47.100353 24329 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:47.101181 24332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:47.101325 24332 catalog_manager.cc:1259] Loaded cluster ID: b1026819a7b44632b2f9eb4d80409e06
I20260925 15:18:47.101562 24332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:47.102355 24332 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:47.102540 24332 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Loaded TSK: 0
I20260925 15:18:47.110879 24332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Generated new TSK 1
I20260925 15:18:47.110982 24332 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:18:47.187069 24325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:47.187342 24325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:47.187389 24325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:47.188661 24325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:47.188731 24325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:47.188782 24325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:18:47.190541 24325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:40599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=45731
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:47.190832 24325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:47.191079 24325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:47.197074 24353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:47.197054 24351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:47.197057 24350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:47.197369 24325 server_base.cc:1034] running on GCE node
I20260925 15:18:47.197574 24325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:47.197858 24325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:47.199033 24325 hybrid_clock.cc:648] HybridClock initialized: now 1790349527199020 us; error 28 us; skew 500 ppm
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:47.201866 24325 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:18:47.202351 24325 webserver.cc:460] Webserver started at http://127.8.43.129:45731/ using document root <none> and password file <none>
I20260925 15:18:47.202600 24325 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:47.202652 24325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:47.204064 24325 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:47.204784 24360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:47.204954 24325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:47.205055 24325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "2d67a3639df3470a80d14e735ca63a2f"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "74a6f90d8b0056ea4dac61d4fe9a4f14"
server_key_iv: "5ec3cff8ae687f26ca699b05311e3192"
server_key_version: "encryptionkey@0"
I20260925 15:18:47.205407 24325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:47.215683 24325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:47.217832 24325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:47.217970 24325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:47.218226 24325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:47.218739 24367 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:18:47.220252 24325 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:18:47.220301 24325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:18:47.220336 24325 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:18:47.220985 24325 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:18:47.221029 24325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:18:47.221071 24367 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Bootstrap starting.
I20260925 15:18:47.228683 24325 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:40599
I20260925 15:18:47.229198 24325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:18:47.229760 24474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:40599 every 8 connection(s)
I20260925 15:18:47.232411 24367 log.cc:824] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:47.234937 24367 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:18:47.235260 24367 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Bootstrap complete.
I20260925 15:18:47.236188 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 24325
I20260925 15:18:47.236312 24367 ts_tablet_manager.cc:1397] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260925 15:18:47.236474 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:32993
--local_ip_for_outbound_sockets=127.8.43.130
--tserver_master_addrs=127.8.43.190:44753
--webserver_port=34149
--webserver_interface=127.8.43.130
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:47.238018 24367 raft_consensus.cc:348] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.238370 24367 raft_consensus.cc:729] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d67a3639df3470a80d14e735ca63a2f, State: Initialized, Role: FOLLOWER
I20260925 15:18:47.238526 24367 consensus_queue.cc:260] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:47.238801 24367 ts_tablet_manager.cc:1428] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260925 15:18:47.243057 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.229949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:54867)
0925 15:18:47.230201 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.230241 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:47.230907 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:47.231632 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:47.231639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:47.232028 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:47.232331 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.232338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.234216 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.234217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.235507 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.235510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.236311 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.236312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.236680 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.237377 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:47.237389 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:47.239129 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:47.241539 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.241545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.241546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.241757 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.242044 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.242045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.242045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.242139 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.242610 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:47.242614 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:47.242720 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 15:18:47.242797 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:18:47.242923 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
I20260925 15:18:47.243340 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.230041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:54867 (local address 127.8.43.190:44753)
0925 15:18:47.231134 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.231143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.231183 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:18:47.231350 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.231357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.231414 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.231548 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:47.232463 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.234084 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.235593 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.236166 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.237958 (+ 1792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.239294 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:47.239298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:47.239299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:47.239311 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:47.241446 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.241841 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.241845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.241845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.241969 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.242214 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.242215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.242215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.242534 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:47.242634 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.242891 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.243043 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":88,"threads_started":1}
I20260925 15:18:47.243954 24475 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44753
I20260925 15:18:47.244057 24475 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:47.244259 24475 heartbeater.cc:511] Master 127.8.43.190:44753 requested a full tablet report, sending...
I20260925 15:18:47.245030 24287 ts_manager.cc:194] Registered new tserver with Master: 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599)
I20260925 15:18:47.246193 24287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:54867
I20260925 15:18:47.247172 24475 heartbeater.cc:503] Master 127.8.43.190:44753 was elected leader, sending a full tablet report...
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.129@KRBTEST.COM
I20260925 15:18:47.254227 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.245820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.8.43.129:40599)
0925 15:18:47.246110 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.246114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.246124 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:47.246155 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.246157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.246171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.246248 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:47.246791 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.247840 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.249020 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.249287 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.249339 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.251393 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:47.251400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:47.251401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:47.251407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:47.252962 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.253371 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.253374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.253374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.253434 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.253637 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.253638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.253638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.253777 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:47.253876 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.254046 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.254132 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":93,"threads_started":1}
I20260925 15:18:47.254227 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.245634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:40599 (local address 127.0.0.1:45570)
0925 15:18:47.245776 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.245816 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:47.245902 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:47.246265 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:47.246271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:47.246641 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:47.246705 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.246707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.247914 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.247915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.248936 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.248940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.249357 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.249358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.249437 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.249863 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:47.249869 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:47.251293 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:47.253064 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.253072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.253074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.253272 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.253502 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.253503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.253504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.253564 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.253849 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:47.253856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:47.253906 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:18:47.254009 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:18:47.254130 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":90,"threads_started":1}
I20260925 15:18:47.254628 24394 tablet_service.cc:1514] Processing DeleteTablet for tablet 03151cc11bc745869ac30a2f69c8d8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:26 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:45570
I20260925 15:18:47.255134 24484 tablet_replica.cc:330] stopping tablet replica
I20260925 15:18:47.255302 24484 raft_consensus.cc:2229] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:47.255373 24484 raft_consensus.cc:2258] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:47.255676 24484 ts_tablet_manager.cc:1905] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:47.258128 24484 ts_tablet_manager.cc:1918] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:47.258198 24484 log.cc:1196] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal/wals/03151cc11bc745869ac30a2f69c8d8c2
I20260925 15:18:47.258453 24484 ts_tablet_manager.cc:1939] T 03151cc11bc745869ac30a2f69c8d8c2 P 2d67a3639df3470a80d14e735ca63a2f: Deleting consensus metadata
I20260925 15:18:47.258973 24272 catalog_manager.cc:4841] TS 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599): tablet 03151cc11bc745869ac30a2f69c8d8c2 (table test-table [id=7587553968664186b823a880ca665e2d]) successfully deleted
W20260925 15:18:47.324021 24480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:47.324208 24480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:47.324229 24480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:47.325372 24480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:47.325413 24480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:47.325440 24480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:18:47.326865 24480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:32993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=34149
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:47.327090 24480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:47.327353 24480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:47.332141 24492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:47.332142 24490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:47.332147 24489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:47.332480 24480 server_base.cc:1034] running on GCE node
I20260925 15:18:47.332705 24480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:47.332996 24480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:47.334154 24480 hybrid_clock.cc:648] HybridClock initialized: now 1790349527334144 us; error 44 us; skew 500 ppm
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:47.336552 24480 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:18:47.337021 24480 webserver.cc:460] Webserver started at http://127.8.43.130:34149/ using document root <none> and password file <none>
I20260925 15:18:47.337244 24480 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:47.337323 24480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:47.338632 24480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:47.339457 24499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:47.339643 24480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:47.339757 24480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "ebcec57e0f8644a5a5e2ad7607dffdce"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "adc75735aa934988b220c6096757f989"
server_key_iv: "16b491cf2a824bd5766f0692918a5d1b"
server_key_version: "encryptionkey@0"
I20260925 15:18:47.340091 24480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:47.362720 24480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:47.364631 24480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:47.364801 24480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:47.365022 24480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:47.365447 24506 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:18:47.366881 24480 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:18:47.366940 24480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:18:47.366982 24480 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:18:47.367615 24480 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:18:47.367669 24480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:18:47.367749 24506 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Bootstrap starting.
I20260925 15:18:47.369277 24506 log.cc:824] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:47.372699 24506 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:18:47.373015 24506 tablet_bootstrap.cc:492] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Bootstrap complete.
I20260925 15:18:47.373261 24506 ts_tablet_manager.cc:1397] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 15:18:47.374423 24480 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:32993
I20260925 15:18:47.374509 24614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:32993 every 8 connection(s)
I20260925 15:18:47.374811 24480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:18:47.375409 24506 raft_consensus.cc:348] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.375833 24506 raft_consensus.cc:729] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcec57e0f8644a5a5e2ad7607dffdce, State: Initialized, Role: FOLLOWER
I20260925 15:18:47.375969 24506 consensus_queue.cc:260] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.376232 24506 ts_tablet_manager.cc:1428] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:47.383399 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 24480
I20260925 15:18:47.383576 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.375537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:42933 (local address 127.8.43.190:44753)
0925 15:18:47.375631 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.375633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.375880 (+ 247us) server_negotiation.cc:407] Connection header received
0925 15:18:47.376027 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.376028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.376039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.376108 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:47.376790 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.377291 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.378215 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.378414 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.378463 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.380940 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:47.380941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:47.380942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:47.380948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:47.382365 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.382689 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.382690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.382690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.382747 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.382950 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.382951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.382951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.383046 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:47.383094 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.383392 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.383483 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:18:47.383668 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:39127
--local_ip_for_outbound_sockets=127.8.43.131
--tserver_master_addrs=127.8.43.190:44753
--webserver_port=46081
--webserver_interface=127.8.43.131
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:47.383891 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.375500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:42933)
0925 15:18:47.375678 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.375727 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:47.375907 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:47.376216 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:47.376219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:47.376524 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:47.376706 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.376717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.377382 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.377382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.378134 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.378135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.378483 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.378483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.378577 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.379023 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:47.379031 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:47.380857 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:47.382432 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.382437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.382438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.382628 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.382803 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.382803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.382804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.382897 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.383098 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:47.383102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:47.383229 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 15:18:47.383379 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 15:18:47.383552 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260925 15:18:47.384436 24615 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44753
I20260925 15:18:47.384547 24615 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:47.384709 24615 heartbeater.cc:511] Master 127.8.43.190:44753 requested a full tablet report, sending...
I20260925 15:18:47.385318 24287 ts_manager.cc:194] Registered new tserver with Master: ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993)
I20260925 15:18:47.385972 24287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:42933
I20260925 15:18:47.386792 24615 heartbeater.cc:503] Master 127.8.43.190:44753 was elected leader, sending a full tablet report...
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.130@KRBTEST.COM
I20260925 15:18:47.393244 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.385631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:32993 (local address 127.0.0.1:60856)
0925 15:18:47.385738 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.385749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:47.385786 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:47.386055 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:47.386055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:47.386222 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:47.386280 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.386282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.387499 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.387500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.387995 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.387996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.388509 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.388510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.388590 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.389008 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:47.389013 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:47.390380 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:47.392197 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.392198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.392198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.392371 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.392602 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.392603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.392603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.392665 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.392949 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:47.392950 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:47.392991 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:18:47.393056 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:18:47.393152 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 15:18:47.393272 24620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.385756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.8.43.130:32993)
0925 15:18:47.385908 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.385913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.385921 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:18:47.385951 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.385953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.385966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.386357 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:47.386399 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.387418 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.388099 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.388418 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.388508 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.390480 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:47.390487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:47.390487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:47.390494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:47.392078 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.392463 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.392466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.392467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.392524 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.392744 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.392745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.392745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.392876 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:47.392969 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.393084 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.393166 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260925 15:18:47.393828 24549 tablet_service.cc:1514] Processing DeleteTablet for tablet 03151cc11bc745869ac30a2f69c8d8c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:26 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:60856
I20260925 15:18:47.394345 24621 tablet_replica.cc:330] stopping tablet replica
I20260925 15:18:47.394461 24621 raft_consensus.cc:2229] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:47.394546 24621 raft_consensus.cc:2258] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:47.394824 24621 ts_tablet_manager.cc:1905] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:47.397092 24621 ts_tablet_manager.cc:1918] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:47.397178 24621 log.cc:1196] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/03151cc11bc745869ac30a2f69c8d8c2
I20260925 15:18:47.397436 24621 ts_tablet_manager.cc:1939] T 03151cc11bc745869ac30a2f69c8d8c2 P ebcec57e0f8644a5a5e2ad7607dffdce: Deleting consensus metadata
I20260925 15:18:47.397912 24273 catalog_manager.cc:4841] TS ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993): tablet 03151cc11bc745869ac30a2f69c8d8c2 (table test-table [id=7587553968664186b823a880ca665e2d]) successfully deleted
W20260925 15:18:47.465615 24619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:47.465910 24619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:47.465957 24619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:47.467295 24619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:47.467346 24619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:47.467374 24619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:18:47.468798 24619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:39127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=46081
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:18:47.469022 24619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:47.469269 24619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:47.474159 24629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:47.474190 24627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:47.474334 24619 server_base.cc:1034] running on GCE node
W20260925 15:18:47.474159 24626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:47.474560 24619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:47.474803 24619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:47.475980 24619 hybrid_clock.cc:648] HybridClock initialized: now 1790349527475964 us; error 32 us; skew 500 ppm
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:47.478358 24619 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:18:47.478809 24619 webserver.cc:460] Webserver started at http://127.8.43.131:46081/ using document root <none> and password file <none>
I20260925 15:18:47.478995 24619 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:47.479045 24619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:47.480437 24619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:47.480960 24636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:47.481138 24619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:18:47.481236 24619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "d87a1123f3384529bbacdc62ce577705"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "5c8de14c2f4947d792ee89d655e41ed2"
server_key_iv: "648adf8110631e43c5bbab518de9195e"
server_key_version: "encryptionkey@0"
I20260925 15:18:47.481527 24619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:47.489995 24619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:47.492244 24619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:47.492345 24619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:47.492503 24619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:18:47.492723 24619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:18:47.492753 24619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:47.492774 24619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:18:47.492786 24619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:47.498147 24619 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:39127
I20260925 15:18:47.498237 24749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:39127 every 8 connection(s)
I20260925 15:18:47.498560 24619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:18:47 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:18:47.507831 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.499303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:53373 (local address 127.8.43.190:44753)
0925 15:18:47.499368 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.499370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.499562 (+ 192us) server_negotiation.cc:407] Connection header received
0925 15:18:47.500165 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.500167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.500177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.500240 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:47.501191 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.501677 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.502731 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.502919 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.502980 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.505223 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:47.505224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:47.505224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:47.505231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:47.506608 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.506956 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.506957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.506958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.507008 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.507276 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.507277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.507277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.507414 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:47.507479 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.507709 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.507770 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:18:47.507995 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.499289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:53373)
0925 15:18:47.499445 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.499478 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:47.500059 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:47.500294 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:47.500297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:47.500831 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:47.501115 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.501120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.501783 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.501792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.502592 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.502594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.503001 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.503002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.503102 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.503730 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:47.503739 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:47.505136 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:47.506679 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.506684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.506685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.506891 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.507066 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:47.507066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:47.507067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:47.507156 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:47.507484 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:47.507488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:47.507595 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 15:18:47.507677 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:18:47.507794 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260925 15:18:47.508512 24750 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44753
I20260925 15:18:47.508611 24750 heartbeater.cc:464] Registering TS with master...
I20260925 15:18:47.508699 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 24619
I20260925 15:18:47.508829 24750 heartbeater.cc:511] Master 127.8.43.190:44753 requested a full tablet report, sending...
I20260925 15:18:47.509254 24287 ts_manager.cc:194] Registered new tserver with Master: d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127)
I20260925 15:18:47.509764 24287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:53373
I20260925 15:18:47.515475 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.509104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.8.43.190:44753)
0925 15:18:47.509143 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.509145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.509320 (+ 175us) server_negotiation.cc:407] Connection header received
0925 15:18:47.509364 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.509365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.509373 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.509425 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:47.510140 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.510590 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.511444 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.511646 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.511719 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.512930 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:47.512931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:47.512932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:47.512939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:47.514338 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.514712 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.514714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.514714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.514771 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.514966 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.514967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.514967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.515071 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:47.515283 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.515320 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.515383 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:18:47.516179 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:18:47.518534 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.516499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.8.43.190:44753)
0925 15:18:47.516554 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.516555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.516662 (+ 107us) server_negotiation.cc:407] Connection header received
0925 15:18:47.516691 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.516692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.516741 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.516802 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:47.516892 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.517298 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.517973 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.518138 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.518219 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.518343 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:47.518380 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:18:47.518496 (+ 116us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42218: token expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 15:18:47.518599 24478 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42218: token expired
W20260925 15:18:47.518683 24218 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.43.190:44753)
I20260925 15:18:47.524334 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.518938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.8.43.190:44753)
0925 15:18:47.518973 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.518974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.518984 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:47.519050 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.519051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.519059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.519110 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:47.519629 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.520059 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.520749 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.520929 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.521010 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.522030 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:47.522031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:47.522032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:47.522039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:47.523261 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.523603 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.523604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.523605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.523662 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:47.523859 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:47.523860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:47.523860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:47.523963 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:47.524066 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.524166 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.524243 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:18:47.525339 24287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:18:47.526942 24287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:18:47.529036 24549 tablet_service.cc:1467] Processing CreateTablet for tablet dd171412fd8143c7adc6f90e2ed567ba (DEFAULT_TABLE table=test-table [id=1dd3ab79de504e9fbec7991c7417aeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:47.529114 24394 tablet_service.cc:1467] Processing CreateTablet for tablet dd171412fd8143c7adc6f90e2ed567ba (DEFAULT_TABLE table=test-table [id=1dd3ab79de504e9fbec7991c7417aeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:47.529244 24549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd171412fd8143c7adc6f90e2ed567ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:47.529249 24394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd171412fd8143c7adc6f90e2ed567ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:47.530828 24506 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: Bootstrap starting.
I20260925 15:18:47.531047 24367 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Bootstrap starting.
I20260925 15:18:47.531277 24506 tablet_bootstrap.cc:654] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:47.531414 24367 tablet_bootstrap.cc:654] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:47.531958 24506 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: No bootstrap required, opened a new log
I20260925 15:18:47.532011 24506 ts_tablet_manager.cc:1397] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:18:47.532318 24506 raft_consensus.cc:348] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.532400 24506 raft_consensus.cc:374] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:47.532451 24506 raft_consensus.cc:729] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcec57e0f8644a5a5e2ad7607dffdce, State: Initialized, Role: FOLLOWER
I20260925 15:18:47.532446 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.528837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:39127 (local address 127.0.0.1:57286)
0925 15:18:47.528886 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.528898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:47.528937 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:47.529319 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:47.529320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:47.529322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:47.529364 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.529365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.530258 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.530259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.531115 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.531117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.532198 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.532199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.532261 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.532282 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:18:47.532341 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:18:47.532382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:18:47.532466 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.528959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.8.43.131:39127)
0925 15:18:47.529148 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.529152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.529162 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:18:47.529205 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.529207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.529221 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.529317 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:47.529426 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.530174 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.531273 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.532099 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.532197 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.532266 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.532356 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.532391 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20260925 15:18:47.532497 24506 consensus_queue.cc:260] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.532614 24506 ts_tablet_manager.cc:1428] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:18:47.532821 24367 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: No bootstrap required, opened a new log
I20260925 15:18:47.532871 24367 ts_tablet_manager.cc:1397] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:18:47.533244 24367 raft_consensus.cc:348] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.533201 24684 tablet_service.cc:1467] Processing CreateTablet for tablet dd171412fd8143c7adc6f90e2ed567ba (DEFAULT_TABLE table=test-table [id=1dd3ab79de504e9fbec7991c7417aeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:18:47.533335 24367 raft_consensus.cc:374] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:47.533387 24367 raft_consensus.cc:729] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d67a3639df3470a80d14e735ca63a2f, State: Initialized, Role: FOLLOWER
I20260925 15:18:47.533432 24367 consensus_queue.cc:260] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.533541 24367 ts_tablet_manager.cc:1428] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:18:47.533566 24684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd171412fd8143c7adc6f90e2ed567ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:47.536365 24758 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Bootstrap starting.
I20260925 15:18:47.536985 24758 tablet_bootstrap.cc:654] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:47.537485 24758 log.cc:824] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:47.538146 24758 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: No bootstrap required, opened a new log
I20260925 15:18:47.538199 24758 ts_tablet_manager.cc:1397] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:18:47.539757 24758 raft_consensus.cc:348] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.539920 24758 raft_consensus.cc:374] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:47.539942 24758 raft_consensus.cc:729] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87a1123f3384529bbacdc62ce577705, State: Initialized, Role: FOLLOWER
I20260925 15:18:47.540073 24758 consensus_queue.cc:260] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.540309 24758 ts_tablet_manager.cc:1428] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:18:47.540340 24750 heartbeater.cc:503] Master 127.8.43.190:44753 was elected leader, sending a full tablet report...
W20260925 15:18:47.625372 24616 tablet.cc:2369] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:18:47.730020 24476 tablet.cc:2369] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:18:47.749112 24751 tablet.cc:2369] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:18:47.792683 24618 raft_consensus.cc:482] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:18:47.792841 24618 raft_consensus.cc:504] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.793314 24618 leader_election.cc:288] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127), 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599)
I20260925 15:18:47.797586 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.793658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:46139 (local address 127.8.43.129:40599)
0925 15:18:47.793800 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.793802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.793993 (+ 191us) server_negotiation.cc:407] Connection header received
0925 15:18:47.794055 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.794055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.794078 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.794172 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:47.794326 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.795029 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.796550 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.797195 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.797270 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.797354 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.797467 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.797515 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260925 15:18:47.797585 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.793591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:34679 (local address 127.8.43.131:39127)
0925 15:18:47.793648 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.793650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.793662 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:18:47.793789 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.793790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.793801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.793872 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:18:47.794158 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.794954 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.796550 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.797175 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.797269 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.797279 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.797467 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.797512 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 15:18:47.797600 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.793537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:39127 (local address 127.8.43.130:34679)
0925 15:18:47.793598 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.793613 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:47.793680 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:47.793951 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:47.793952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:47.793955 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:47.794055 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.794057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.795047 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.795048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.796418 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.796422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.797276 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.797277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.797357 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.797381 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:18:47.797442 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:18:47.797491 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":42}
I20260925 15:18:47.797617 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.793543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:40599 (local address 127.8.43.130:46139)
0925 15:18:47.793862 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.793876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:47.793988 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:47.794201 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:47.794201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:47.794203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:18:47.794256 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.794256 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.795122 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.795123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.796418 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:47.796422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.797294 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.797295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:47.797358 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.797381 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:18:47.797442 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:18:47.797491 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":53,"thread_start_us":131,"threads_started":1}
I20260925 15:18:47.798013 24704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd171412fd8143c7adc6f90e2ed567ba" candidate_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87a1123f3384529bbacdc62ce577705" is_pre_election: true
I20260925 15:18:47.798194 24704 raft_consensus.cc:2454] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebcec57e0f8644a5a5e2ad7607dffdce in term 0.
I20260925 15:18:47.798194 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd171412fd8143c7adc6f90e2ed567ba" candidate_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d67a3639df3470a80d14e735ca63a2f" is_pre_election: true
I20260925 15:18:47.798373 24419 raft_consensus.cc:2454] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebcec57e0f8644a5a5e2ad7607dffdce in term 0.
I20260925 15:18:47.798480 24503 leader_election.cc:302] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d87a1123f3384529bbacdc62ce577705, ebcec57e0f8644a5a5e2ad7607dffdce; no voters:
I20260925 15:18:47.798605 24618 raft_consensus.cc:2781] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:18:47.798659 24618 raft_consensus.cc:482] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:18:47.798676 24618 raft_consensus.cc:3037] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:47.799381 24618 raft_consensus.cc:504] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.799520 24618 leader_election.cc:288] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [CANDIDATE]: Term 1 election: Requested vote from peers d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127), 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599)
I20260925 15:18:47.799682 24704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd171412fd8143c7adc6f90e2ed567ba" candidate_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87a1123f3384529bbacdc62ce577705"
I20260925 15:18:47.799787 24704 raft_consensus.cc:3037] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:47.799765 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd171412fd8143c7adc6f90e2ed567ba" candidate_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d67a3639df3470a80d14e735ca63a2f"
I20260925 15:18:47.799872 24419 raft_consensus.cc:3037] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:47.800534 24419 raft_consensus.cc:2454] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebcec57e0f8644a5a5e2ad7607dffdce in term 1.
I20260925 15:18:47.800555 24704 raft_consensus.cc:2454] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebcec57e0f8644a5a5e2ad7607dffdce in term 1.
I20260925 15:18:47.800752 24501 leader_election.cc:302] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d67a3639df3470a80d14e735ca63a2f, ebcec57e0f8644a5a5e2ad7607dffdce; no voters:
I20260925 15:18:47.800863 24618 raft_consensus.cc:2781] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:47.801016 24618 raft_consensus.cc:686] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 LEADER]: Becoming Leader. State: Replica: ebcec57e0f8644a5a5e2ad7607dffdce, State: Running, Role: LEADER
I20260925 15:18:47.801112 24618 consensus_queue.cc:237] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:18:47.801942 24287 catalog_manager.cc:5495] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce reported cstate change: term changed from 0 to 1, leader changed from <none> to ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130). New cstate: current_term: 1 leader_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } health_report { overall_health: UNKNOWN } } }
I20260925 15:18:47.817130 24620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.814854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.8.43.130:32993)
0925 15:18:47.814923 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:47.814925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:47.814932 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:18:47.814962 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:47.814963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:47.814979 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:47.815062 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:47.815188 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.815719 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.816408 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:47.816607 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:47.816679 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:47.816857 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:47.816948 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:47.817026 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:47.817062 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 15:18:47.817079 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:18:47.818539 24704 raft_consensus.cc:1261] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 1 FOLLOWER]: Refusing update from remote peer ebcec57e0f8644a5a5e2ad7607dffdce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:47.818540 24419 raft_consensus.cc:1261] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Refusing update from remote peer ebcec57e0f8644a5a5e2ad7607dffdce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:47.818912 24618 consensus_queue.cc:1035] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:47.819047 24763 consensus_queue.cc:1035] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:47.820749 24765 mvcc.cc:205] Tried to move back new op lower bound from 7333271665942286336 to 7333271665873993728. Current Snapshot: MvccSnapshot[applied={T|T < 7333271665942286336}]
I20260925 15:18:47.821080 24766 mvcc.cc:205] Tried to move back new op lower bound from 7333271665942286336 to 7333271665873993728. Current Snapshot: MvccSnapshot[applied={T|T < 7333271665942286336}]
I20260925 15:18:47.822820 24769 mvcc.cc:205] Tried to move back new op lower bound from 7333271665942286336 to 7333271665873993728. Current Snapshot: MvccSnapshot[applied={T|T < 7333271665942286336 or (T in {7333271665942286336})}]
I20260925 15:18:47.823071 24286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:47.823159 24286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:18:47.823941 24286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Sending DeleteTablet for 3 replicas of tablet dd171412fd8143c7adc6f90e2ed567ba
I20260925 15:18:47.824247 24684 tablet_service.cc:1514] Processing DeleteTablet for tablet dd171412fd8143c7adc6f90e2ed567ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:47 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:57286
I20260925 15:18:47.824273 24394 tablet_service.cc:1514] Processing DeleteTablet for tablet dd171412fd8143c7adc6f90e2ed567ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:47 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:45570
I20260925 15:18:47.824373 24549 tablet_service.cc:1514] Processing DeleteTablet for tablet dd171412fd8143c7adc6f90e2ed567ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:18:47 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:60856
I20260925 15:18:47.824452 24621 tablet_replica.cc:330] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: stopping tablet replica
I20260925 15:18:47.824506 24621 raft_consensus.cc:2229] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:18:47.824590 24621 raft_consensus.cc:2258] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:47.824864 24621 ts_tablet_manager.cc:1905] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:47.824914 24776 tablet_replica.cc:330] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: stopping tablet replica
I20260925 15:18:47.824922 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 24325
I20260925 15:18:47.824999 24776 raft_consensus.cc:2229] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:18:47.825088 24776 raft_consensus.cc:2258] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:18:47.825517 24776 ts_tablet_manager.cc:1905] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:18:47.826602 24621 ts_tablet_manager.cc:1918] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:47.826653 24621 log.cc:1196] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/dd171412fd8143c7adc6f90e2ed567ba
I20260925 15:18:47.826879 24621 ts_tablet_manager.cc:1939] T dd171412fd8143c7adc6f90e2ed567ba P ebcec57e0f8644a5a5e2ad7607dffdce: Deleting consensus metadata
I20260925 15:18:47.827064 24776 ts_tablet_manager.cc:1918] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:18:47.827121 24776 log.cc:1196] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/dd171412fd8143c7adc6f90e2ed567ba
I20260925 15:18:47.827409 24776 ts_tablet_manager.cc:1939] T dd171412fd8143c7adc6f90e2ed567ba P d87a1123f3384529bbacdc62ce577705: Deleting consensus metadata
I20260925 15:18:47.827380 24273 catalog_manager.cc:4841] TS ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993): tablet dd171412fd8143c7adc6f90e2ed567ba (table test-table [id=1dd3ab79de504e9fbec7991c7417aeb4]) successfully deleted
W20260925 15:18:47.831341 24272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:40599 (error 108)
I20260925 15:18:47.831722 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 24480
I20260925 15:18:47.831830 24274 catalog_manager.cc:4841] TS d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127): tablet dd171412fd8143c7adc6f90e2ed567ba (table test-table [id=1dd3ab79de504e9fbec7991c7417aeb4]) successfully deleted
I20260925 15:18:47.831846 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:47.831701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:40599 (local address 127.0.0.1:45576)
0925 15:18:47.831768 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:47.831810 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:40599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:18:47.832692 24272 catalog_manager.cc:4584] TS 2d67a3639df3470a80d14e735ca63a2f (127.8.43.129:40599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd171412fd8143c7adc6f90e2ed567ba: Network error: Client connection negotiation failed: client connection to 127.8.43.129:40599: connect: Connection refused (error 111)
I20260925 15:18:47.836637 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 24619
I20260925 15:18:47.841938 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 24254
I20260925 15:19:07.847329 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44753
--webserver_interface=127.8.43.190
--webserver_port=32961
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:07.931783 24778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:07.931988 24778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:07.932009 24778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:07.933252 24778 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:19:07.933296 24778 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:19:07.933310 24778 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:19:07.933321 24778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:07.933336 24778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:07.933355 24778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:07.933403 24778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:07.933471 24778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:07.934993 24778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=32961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:07.935276 24778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:07.935536 24778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:07.940743 24786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:07.940771 24783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:07.940739 24778 server_base.cc:1034] running on GCE node
W20260925 15:19:07.940758 24784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:07.941303 24778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:07.941677 24778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:07.942843 24778 hybrid_clock.cc:648] HybridClock initialized: now 1790349547942829 us; error 30 us; skew 500 ppm
Sep 25 15:19:07 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349547, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:07.950979 24778 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:07.951468 24778 webserver.cc:460] Webserver started at http://127.8.43.190:32961/ using document root <none> and password file <none>
I20260925 15:19:07.951704 24778 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:07.951771 24778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:07.953125 24778 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:07.955209 24793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:07.955449 24778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:07.955554 24778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "c99dcc23f0dc47848037ff660f7ed64d"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "fb7357d990e6928909ab93f89a2205fc"
server_key_iv: "90baa3c04afc2db869e2021a61ead91c"
server_key_version: "encryptionkey@0"
I20260925 15:19:07.955964 24778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:07.968106 24778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:07.969976 24778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:07.970135 24778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:07.974318 24778 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44753
I20260925 15:19:07.974368 24845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44753 every 8 connection(s)
I20260925 15:19:07.974749 24778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:07.976266 24846 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:19:07.976933 24846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Bootstrap starting.
I20260925 15:19:07.982298 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 24778
I20260925 15:19:07.982652 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:40599
--local_ip_for_outbound_sockets=127.8.43.129
--tserver_master_addrs=127.8.43.190:44753
--webserver_port=45731
--webserver_interface=127.8.43.129
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:07.985854 24846 log.cc:824] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:07.988440 24846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:19:07.988772 24846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Bootstrap complete.
I20260925 15:19:07.991003 24846 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:19:07.991266 24846 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99dcc23f0dc47848037ff660f7ed64d, State: Initialized, Role: FOLLOWER
I20260925 15:19:07.991456 24846 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:19:07.991554 24846 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:07.991606 24846 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:07.991660 24846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:19:07.992651 24846 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:19:07.992771 24846 leader_election.cc:302] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99dcc23f0dc47848037ff660f7ed64d; no voters:
I20260925 15:19:07.992974 24846 leader_election.cc:288] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:19:07.993011 24850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:19:07.993126 24850 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [term 3 LEADER]: Becoming Leader. State: Replica: c99dcc23f0dc47848037ff660f7ed64d, State: Running, Role: LEADER
I20260925 15:19:07.993280 24846 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:07.993280 24850 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } }
I20260925 15:19:07.993707 24851 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c99dcc23f0dc47848037ff660f7ed64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } } }
I20260925 15:19:07.993794 24851 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:07.994123 24852 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99dcc23f0dc47848037ff660f7ed64d. Latest consensus state: current_term: 3 leader_uuid: "c99dcc23f0dc47848037ff660f7ed64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99dcc23f0dc47848037ff660f7ed64d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44753 } } }
I20260925 15:19:07.994189 24852 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:07.994349 24857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:07.995265 24857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:07.995445 24857 catalog_manager.cc:1259] Loaded cluster ID: b1026819a7b44632b2f9eb4d80409e06
I20260925 15:19:07.995658 24857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:07.996585 24857 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:07.996776 24857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Loaded TSK: 0
I20260925 15:19:07.996831 24857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c99dcc23f0dc47848037ff660f7ed64d: Loaded TSK: 1
F20260925 15:19:07.997084 24857 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff33df799ad google::LogMessage::Fail() at ??:0
@ 0x7ff33df7c88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff33df794de google::LogMessage::Flush() at ??:0
@ 0x7ff33df7aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff3406e6104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff3406e3d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff345ed31cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff345ed348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff345ed3594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff345e99c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff345ea3ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff33fe470d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff33fe3c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff340ca06db start_thread at ??:0
@ 0x7ff33ca7761f clone at ??:0
W20260925 15:19:08.081096 24848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:08.081307 24848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:08.081352 24848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:08.082504 24848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:08.082568 24848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:08.082617 24848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:19:08.084146 24848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:40599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=45731
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:08.084408 24848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:08.084631 24848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:08.089411 24890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:08.089403 24888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:08.089619 24848 server_base.cc:1034] running on GCE node
W20260925 15:19:08.089439 24887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:08.089983 24848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:08.090257 24848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:08.091419 24848 hybrid_clock.cc:648] HybridClock initialized: now 1790349548091407 us; error 31 us; skew 500 ppm
Sep 25 15:19:08 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349548, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:08.093955 24848 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:19:08.094413 24848 webserver.cc:460] Webserver started at http://127.8.43.129:45731/ using document root <none> and password file <none>
I20260925 15:19:08.094643 24848 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:08.094707 24848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:08.096060 24848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:08.096748 24897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:08.096933 24848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:08.097026 24848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "2d67a3639df3470a80d14e735ca63a2f"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "74a6f90d8b0056ea4dac61d4fe9a4f14"
server_key_iv: "5ec3cff8ae687f26ca699b05311e3192"
server_key_version: "encryptionkey@0"
I20260925 15:19:08.097342 24848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:08.114775 24848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:08.116986 24848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:08.117143 24848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:08.117373 24848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:08.117835 24904 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:19:08.119940 24848 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:19:08.119997 24848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:19:08.120054 24848 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:19:08.120690 24848 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:19:08.120746 24848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:19:08.120817 24904 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Bootstrap starting.
I20260925 15:19:08.127719 24848 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:40599
I20260925 15:19:08.127800 25011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:40599 every 8 connection(s)
I20260925 15:19:08.128196 24848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:08.131781 24904 log.cc:824] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:08.134640 24904 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:19:08.134955 24904 tablet_bootstrap.cc:492] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Bootstrap complete.
I20260925 15:19:08.136153 24904 ts_tablet_manager.cc:1397] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260925 15:19:08.137545 24904 raft_consensus.cc:348] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:19:08.137871 24904 raft_consensus.cc:729] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d67a3639df3470a80d14e735ca63a2f, State: Initialized, Role: FOLLOWER
I20260925 15:19:08.138005 24904 consensus_queue.cc:260] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:19:08.138221 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 24848
I20260925 15:19:08.138255 24904 ts_tablet_manager.cc:1428] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 15:19:08.138474 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:32993
--local_ip_for_outbound_sockets=127.8.43.130
--tserver_master_addrs=127.8.43.190:44753
--webserver_port=34149
--webserver_interface=127.8.43.130
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:08.195480 25014 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:08.128810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:52623)
0925 15:19:08.128981 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.129023 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:08.129458 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:08.195213 (+ 65755us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
W20260925 15:19:08.195760 25012 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:44753 (0 consecutive failures): Network error: Failed to ping master at 127.8.43.190:44753: Client connection negotiation failed: client connection to 127.8.43.190:44753: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:19:08.196106 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.195988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:48709)
0925 15:19:08.196042 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.196079 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:19:08.196475 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.196391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:53831)
0925 15:19:08.196436 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.196459 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:19:08.196588 25012 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:19:08.219982 25017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:08.220191 25017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:08.220228 25017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:08.221357 25017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:08.221421 25017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:08.221467 25017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:19:08.222882 25017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:32993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=34149
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:08.223124 25017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:08.223384 25017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:08.228734 25023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:08.228803 25017 server_base.cc:1034] running on GCE node
W20260925 15:19:08.228740 25022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:08.228940 25025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:08.229151 25017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:08.229338 25017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:08.230494 25017 hybrid_clock.cc:648] HybridClock initialized: now 1790349548230484 us; error 25 us; skew 500 ppm
Sep 25 15:19:08 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349548, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:08.233063 25017 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:19:08.233496 25017 webserver.cc:460] Webserver started at http://127.8.43.130:34149/ using document root <none> and password file <none>
I20260925 15:19:08.233691 25017 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:08.233731 25017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:08.234958 25017 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:08.235544 25032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:08.235697 25017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:08.235795 25017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "ebcec57e0f8644a5a5e2ad7607dffdce"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "adc75735aa934988b220c6096757f989"
server_key_iv: "16b491cf2a824bd5766f0692918a5d1b"
server_key_version: "encryptionkey@0"
I20260925 15:19:08.236124 25017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:08.246773 25017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:08.248934 25017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:08.249078 25017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:08.249308 25017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:08.249599 25017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:08.249650 25017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:08.249688 25017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:08.249714 25017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:08.255409 25017 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:32993
I20260925 15:19:08.255510 25145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:32993 every 8 connection(s)
I20260925 15:19:08.255806 25017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:08.256872 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.256434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:54593)
0925 15:19:08.256616 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.256687 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
W20260925 15:19:08.257261 25146 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:44753 (0 consecutive failures): Network error: Failed to ping master at 127.8.43.190:44753: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
I20260925 15:19:08.257515 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.257440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:55309)
0925 15:19:08.257470 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.257497 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 15:19:08.257966 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.257838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:52031)
0925 15:19:08.257905 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.257944 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:19:08.258203 25146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:19:08.263512 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 25017
I20260925 15:19:08.263861 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:39127
--local_ip_for_outbound_sockets=127.8.43.131
--tserver_master_addrs=127.8.43.190:44753
--webserver_port=46081
--webserver_interface=127.8.43.131
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:08.345536 25149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:08.345723 25149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:08.345745 25149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:08.346885 25149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:08.346940 25149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:08.346966 25149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:19:08.348420 25149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:39127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=46081
--tserver_master_addrs=127.8.43.190:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:08.348657 25149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:08.348892 25149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:08.353672 25155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:08.353698 25157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:08.354017 25149 server_base.cc:1034] running on GCE node
W20260925 15:19:08.353688 25154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:08.354271 25149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:08.354475 25149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:08.355605 25149 hybrid_clock.cc:648] HybridClock initialized: now 1790349548355594 us; error 31 us; skew 500 ppm
Sep 25 15:19:08 dist-test-slave-c5d6 krb5kdc[23711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349548, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:08.358147 25149 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:19:08.358520 25149 webserver.cc:460] Webserver started at http://127.8.43.131:46081/ using document root <none> and password file <none>
I20260925 15:19:08.358716 25149 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:08.358754 25149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:08.360047 25149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:08.360661 25164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:08.360843 25149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:08.360952 25149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "d87a1123f3384529bbacdc62ce577705"
format_stamp: "Formatted at 2026-09-25 15:18:26 on dist-test-slave-c5d6"
server_key: "5c8de14c2f4947d792ee89d655e41ed2"
server_key_iv: "648adf8110631e43c5bbab518de9195e"
server_key_version: "encryptionkey@0"
I20260925 15:19:08.361263 25149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:08.378312 25149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:08.380716 25149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:08.380872 25149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:08.381098 25149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:08.381390 25149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:08.381441 25149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:08.381484 25149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:08.381510 25149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:08.387120 25149 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:39127
I20260925 15:19:08.387262 25277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:39127 every 8 connection(s)
I20260925 15:19:08.387578 25149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:08.388469 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 25149
I20260925 15:19:08.389256 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.388374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:41181)
0925 15:19:08.388565 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.388654 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
W20260925 15:19:08.389506 25278 heartbeater.cc:644] Failed to heartbeat to 127.8.43.190:44753 (0 consecutive failures): Network error: Failed to ping master at 127.8.43.190:44753: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
I20260925 15:19:08.389904 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.389696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:41875)
0925 15:19:08.389769 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.389880 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:19:08.390290 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:08.390199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:58601)
0925 15:19:08.390242 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:08.390268 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:19:08.390460 25278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:19:09.197204 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:09.196940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:49663)
0925 15:19:09.197101 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:09.197139 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20260925 15:19:09.258755 25283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:09.258526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:52123)
0925 15:19:09.258666 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:09.258699 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260925 15:19:09.390977 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:09.390762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:58761)
0925 15:19:09.390895 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:09.390926 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260925 15:19:09.900046 25285 raft_consensus.cc:482] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:09.900199 25285 raft_consensus.cc:504] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:19:09.900524 25285 leader_election.cc:288] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127), ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993)
Sep 25 15:19:09 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349548, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.130@KRBTEST.COM
Sep 25 15:19:09 dist-test-slave-c5d6 krb5kdc[23711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349548, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.131@KRBTEST.COM
I20260925 15:19:09.910902 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:09.900778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:42209 (local address 127.8.43.130:32993)
0925 15:19:09.900981 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:09.900992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:09.901012 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:19:09.901444 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:09.901446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:09.901489 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:09.901615 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:09.902418 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:09.903430 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:09.904430 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:09.904938 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:09.905055 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:09.907424 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:09.907427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:09.907428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:09.907436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:09.909225 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:09.909718 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:09.909721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:09.909721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:09.909810 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:09.910078 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:09.910079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:09.910080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:09.910297 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:09.910387 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:09.910669 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:19:09.910782 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260925 15:19:09.910887 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:09.900671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:32993 (local address 127.8.43.129:42209)
0925 15:19:09.900792 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:09.900802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:09.900885 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:09.901675 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:09.901678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:09.902068 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:09.902273 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:09.902278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:09.903524 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:09.903525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:09.904338 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:09.904340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:09.905028 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:09.905029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:09.905152 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:09.905821 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:09.905827 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:09.907268 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:09.909335 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:09.909341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:09.909343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:09.909624 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:09.909886 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:09.909887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:09.909887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:09.909999 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:09.910383 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:09.910384 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:09.910514 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 15:19:09.910620 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:19:09.910792 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":55,"thread_start_us":49,"threads_started":1}
I20260925 15:19:09.910945 25289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:09.900810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:36805 (local address 127.8.43.131:39127)
0925 15:19:09.901002 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:09.901011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:09.901028 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:19:09.901484 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:09.901486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:09.901524 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:09.901672 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:09.902399 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:09.903456 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:09.904430 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:09.904947 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:09.905061 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:09.907782 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:09.907785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:09.907786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:09.907796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:09.909281 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:09.909708 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:09.909710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:09.909711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:09.909804 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:09.910079 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:09.910080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:09.910080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:09.910297 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:09.910388 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:09.910680 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:19:09.910824 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260925 15:19:09.911396 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:09.900671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:39127 (local address 127.8.43.129:36805)
0925 15:19:09.900790 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:09.900802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:09.900860 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:09.901686 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:09.901687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:09.902067 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:09.902273 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:09.902278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:09.903562 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:09.903563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:09.904339 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:09.904341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:09.905140 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:09.905141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:09.905181 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:09.905758 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:09.905766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:09.907681 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:09.909370 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:09.909371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:09.909372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:09.909625 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:09.909889 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:09.909890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:09.909891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:09.910003 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:09.910374 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:09.910380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:09.910514 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 15:19:09.910621 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:19:09.910870 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":154,"thread_start_us":66,"threads_started":1}
W20260925 15:19:09.911938 24900 leader_election.cc:341] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993): Not found: Tablet not found: dd171412fd8143c7adc6f90e2ed567ba
W20260925 15:19:09.912029 24901 leader_election.cc:341] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127): Not found: Tablet not found: dd171412fd8143c7adc6f90e2ed567ba
I20260925 15:19:09.912060 24901 leader_election.cc:302] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d67a3639df3470a80d14e735ca63a2f; no voters: d87a1123f3384529bbacdc62ce577705, ebcec57e0f8644a5a5e2ad7607dffdce
I20260925 15:19:09.912145 25285 raft_consensus.cc:2726] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:19:10.197986 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:10.197825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:40755)
0925 15:19:10.197903 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:10.197945 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:19:10.259605 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:10.259365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:48285)
0925 15:19:10.259496 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:10.259554 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260925 15:19:10.391669 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:10.391445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:51955)
0925 15:19:10.391580 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:10.391624 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260925 15:19:11.198866 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.198575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:33993)
0925 15:19:11.198765 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:11.198807 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260925 15:19:11.260320 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.260120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:54119)
0925 15:19:11.260240 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:11.260277 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260925 15:19:11.392344 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:11.392111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:49183)
0925 15:19:11.392258 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:11.392294 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260925 15:19:11.482225 25295 raft_consensus.cc:482] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:11.482343 25295 raft_consensus.cc:504] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:19:11.482579 25295 leader_election.cc:288] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127), ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993)
W20260925 15:19:11.483028 24901 leader_election.cc:341] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127): Not found: Tablet not found: dd171412fd8143c7adc6f90e2ed567ba
W20260925 15:19:11.483167 24900 leader_election.cc:341] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993): Not found: Tablet not found: dd171412fd8143c7adc6f90e2ed567ba
I20260925 15:19:11.483258 24900 leader_election.cc:302] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d67a3639df3470a80d14e735ca63a2f; no voters: d87a1123f3384529bbacdc62ce577705, ebcec57e0f8644a5a5e2ad7607dffdce
I20260925 15:19:11.483328 25295 raft_consensus.cc:2726] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:19:12.199751 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:12.199519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:58761)
0925 15:19:12.199668 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:12.199704 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260925 15:19:12.260964 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:12.260766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:38739)
0925 15:19:12.260889 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:12.260922 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260925 15:19:12.393023 25298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:12.392815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:48409)
0925 15:19:12.392949 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:12.392983 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260925 15:19:13.200516 25299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:13.200269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:52559)
0925 15:19:13.200429 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:13.200470 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":59,"threads_started":1}
I20260925 15:19:13.261710 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:13.261489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:34729)
0925 15:19:13.261631 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:13.261667 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260925 15:19:13.393697 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:13.393461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:55539)
0925 15:19:13.393608 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:13.393652 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":58,"threads_started":1}
I20260925 15:19:14.201357 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:14.201050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:33063)
0925 15:19:14.201243 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:14.201282 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":105,"threads_started":1}
I20260925 15:19:14.262452 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:14.262224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:45633)
0925 15:19:14.262371 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:14.262411 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260925 15:19:14.394583 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:14.394379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:50505)
0925 15:19:14.394509 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:14.394542 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260925 15:19:14.617733 25305 raft_consensus.cc:482] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:14.617821 25305 raft_consensus.cc:504] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:19:14.618032 25305 leader_election.cc:288] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127), ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993)
W20260925 15:19:14.618439 24901 leader_election.cc:341] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127): Not found: Tablet not found: dd171412fd8143c7adc6f90e2ed567ba
W20260925 15:19:14.618510 24900 leader_election.cc:341] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993): Not found: Tablet not found: dd171412fd8143c7adc6f90e2ed567ba
I20260925 15:19:14.618634 24900 leader_election.cc:302] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d67a3639df3470a80d14e735ca63a2f; no voters: d87a1123f3384529bbacdc62ce577705, ebcec57e0f8644a5a5e2ad7607dffdce
I20260925 15:19:14.618711 25305 raft_consensus.cc:2726] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:19:15.202086 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.201845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:40841)
0925 15:19:15.201995 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:15.202036 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260925 15:19:15.263164 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.262936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:33815)
0925 15:19:15.263080 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:15.263120 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260925 15:19:15.395319 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:15.395063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:40969)
0925 15:19:15.395180 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:15.395268 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260925 15:19:16.202842 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:16.202612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:47833)
0925 15:19:16.202753 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:16.202792 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260925 15:19:16.263746 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:16.263538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:49311)
0925 15:19:16.263654 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:16.263685 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260925 15:19:16.396072 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:16.395816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:51973)
0925 15:19:16.395959 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:16.396016 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260925 15:19:17.203616 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:17.203395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:38549)
0925 15:19:17.203526 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:17.203564 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260925 15:19:17.264405 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:17.264184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:53035)
0925 15:19:17.264312 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:17.264354 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260925 15:19:17.396710 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:17.396485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:41995)
0925 15:19:17.396617 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:17.396651 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260925 15:19:17.799649 25315 raft_consensus.cc:482] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:17.799747 25315 raft_consensus.cc:504] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d87a1123f3384529bbacdc62ce577705" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 39127 } } peers { permanent_uuid: "ebcec57e0f8644a5a5e2ad7607dffdce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32993 } } peers { permanent_uuid: "2d67a3639df3470a80d14e735ca63a2f" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 40599 } }
I20260925 15:19:17.800042 25315 leader_election.cc:288] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127), ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993)
W20260925 15:19:17.800468 24901 leader_election.cc:341] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d87a1123f3384529bbacdc62ce577705 (127.8.43.131:39127): Not found: Tablet not found: dd171412fd8143c7adc6f90e2ed567ba
W20260925 15:19:17.800549 24900 leader_election.cc:341] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ebcec57e0f8644a5a5e2ad7607dffdce (127.8.43.130:32993): Not found: Tablet not found: dd171412fd8143c7adc6f90e2ed567ba
I20260925 15:19:17.800612 24900 leader_election.cc:302] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d67a3639df3470a80d14e735ca63a2f; no voters: d87a1123f3384529bbacdc62ce577705, ebcec57e0f8644a5a5e2ad7607dffdce
I20260925 15:19:17.800694 25315 raft_consensus.cc:2726] T dd171412fd8143c7adc6f90e2ed567ba P 2d67a3639df3470a80d14e735ca63a2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:19:18.204339 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.204122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:36387)
0925 15:19:18.204257 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:18.204292 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260925 15:19:18.265192 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.264916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:39653)
0925 15:19:18.265111 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:18.265148 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260925 15:19:18.397423 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:18.397194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:52283)
0925 15:19:18.397341 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:18.397377 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260925 15:19:19.204979 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.204765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:58099)
0925 15:19:19.204887 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:19.204927 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260925 15:19:19.266014 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.265743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:53023)
0925 15:19:19.265912 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:19.265944 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
I20260925 15:19:19.398141 25321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:19.397924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:43601)
0925 15:19:19.398058 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:19.398097 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260925 15:19:20.205781 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:20.205523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:50945)
0925 15:19:20.205689 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:20.205732 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260925 15:19:20.266755 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:20.266534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:39439)
0925 15:19:20.266663 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:20.266707 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260925 15:19:20.398866 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:20.398634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:37953)
0925 15:19:20.398777 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:20.398820 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260925 15:19:21.206645 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.206415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:57863)
0925 15:19:21.206549 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.206592 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260925 15:19:21.267493 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.267287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:46771)
0925 15:19:21.267403 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.267445 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260925 15:19:21.399672 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:21.399443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:34261)
0925 15:19:21.399590 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:21.399626 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260925 15:19:22.207387 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:22.207137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:36395)
0925 15:19:22.207304 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:22.207339 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
I20260925 15:19:22.268273 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:22.268032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:32803)
0925 15:19:22.268171 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:22.268227 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260925 15:19:22.400282 25330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:22.400091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:36505)
0925 15:19:22.400202 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:22.400239 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260925 15:19:23.208177 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:23.207940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.129:44815)
0925 15:19:23.208090 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:23.208131 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260925 15:19:23.269084 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:23.268849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.130:52959)
0925 15:19:23.268992 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:23.269038 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260925 15:19:23.391304 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 24848
I20260925 15:19:23.396646 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 25017
I20260925 15:19:23.400992 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:23.400798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44753 (local address 127.8.43.131:60193)
0925 15:19:23.400917 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:23.400953 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.190:44753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260925 15:19:23.401109 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 25149
2026-09-25T15:19:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58363 ms)
[----------] 30 tests from SecurityITest (256804 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:23 dist-test-slave-c5d6 krb5kdc[25336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:23 dist-test-slave-c5d6 krb5kdc[25336](info): set up 2 sockets
Sep 25 15:19:23 dist-test-slave-c5d6 krb5kdc[25336](info): commencing operation
krb5kdc: starting...
W20260925 15:19:24.442945 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:24 dist-test-slave-c5d6 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:24.559376 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:46251
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:24.638712 25349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:24.638957 25349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:24.638990 25349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:24.640166 25349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:24.640241 25349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:24.640264 25349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:24.640285 25349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:24.640314 25349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:24.641788 25349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:46251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:24.642058 25349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:24.642293 25349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:24.647377 25354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:24.647472 25357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:24.647358 25355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:24.647495 25349 server_base.cc:1034] running on GCE node
I20260925 15:19:24.647853 25349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:24.648125 25349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:24.649312 25349 hybrid_clock.cc:648] HybridClock initialized: now 1790349564649311 us; error 41 us; skew 500 ppm
Sep 25 15:19:24 dist-test-slave-c5d6 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349564, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:24.651852 25349 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:24.652313 25349 webserver.cc:460] Webserver started at http://127.8.43.190:35987/ using document root <none> and password file <none>
I20260925 15:19:24.652556 25349 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:24.652626 25349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:24.652772 25349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:24.653846 25349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "83302fcdfaf248b7b2d7049ce5d84a41"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "c12587e46431236f6b9cf455ed3ef009"
server_key_iv: "ba9ee5f9c2c4796851db457e11829849"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.654219 25349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "83302fcdfaf248b7b2d7049ce5d84a41"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "c12587e46431236f6b9cf455ed3ef009"
server_key_iv: "ba9ee5f9c2c4796851db457e11829849"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.655565 25349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:19:24.656301 25364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:24.656582 25349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:24.656687 25349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "83302fcdfaf248b7b2d7049ce5d84a41"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "c12587e46431236f6b9cf455ed3ef009"
server_key_iv: "ba9ee5f9c2c4796851db457e11829849"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.656759 25349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:24.686889 25349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:24.689358 25349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:24.689538 25349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:24.693790 25349 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:46251
I20260925 15:19:24.693852 25416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:46251 every 8 connection(s)
I20260925 15:19:24.694216 25349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:24.695083 25417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:24.697331 25417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41: Bootstrap starting.
I20260925 15:19:24.697865 25417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:24.698108 25417 log.cc:824] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:24.698738 25417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41: No bootstrap required, opened a new log
I20260925 15:19:24.700294 25417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83302fcdfaf248b7b2d7049ce5d84a41" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46251 } }
I20260925 15:19:24.700428 25417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:24.700452 25417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83302fcdfaf248b7b2d7049ce5d84a41, State: Initialized, Role: FOLLOWER
I20260925 15:19:24.700551 25417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83302fcdfaf248b7b2d7049ce5d84a41" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46251 } }
I20260925 15:19:24.700628 25417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:24.700662 25417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:24.700717 25417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:24.701239 25417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83302fcdfaf248b7b2d7049ce5d84a41" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46251 } }
I20260925 15:19:24.701359 25417 leader_election.cc:302] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83302fcdfaf248b7b2d7049ce5d84a41; no voters:
I20260925 15:19:24.701534 25417 leader_election.cc:288] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:24.701586 25420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:24.701807 25420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [term 1 LEADER]: Becoming Leader. State: Replica: 83302fcdfaf248b7b2d7049ce5d84a41, State: Running, Role: LEADER
I20260925 15:19:24.701819 25417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:24.701915 25420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83302fcdfaf248b7b2d7049ce5d84a41" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46251 } }
I20260925 15:19:24.702312 25422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83302fcdfaf248b7b2d7049ce5d84a41. Latest consensus state: current_term: 1 leader_uuid: "83302fcdfaf248b7b2d7049ce5d84a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83302fcdfaf248b7b2d7049ce5d84a41" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46251 } } }
I20260925 15:19:24.702414 25421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83302fcdfaf248b7b2d7049ce5d84a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83302fcdfaf248b7b2d7049ce5d84a41" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 46251 } } }
I20260925 15:19:24.702466 25422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:24.702514 25421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:24.702999 25429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:24.703516 25429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:24.703967 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 25349
I20260925 15:19:24.704078 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:24.704276 8366 external_mini_cluster.cc:1428] Setting key eb0fadce4e1b094541b6de7fc714da23
I20260925 15:19:24.705142 25429 catalog_manager.cc:1347] Generated new cluster ID: 224791cb2eb2499e80b24273012cc7e0
I20260925 15:19:24.705204 25429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:19:24 dist-test-slave-c5d6 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:24.716001 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:24.704736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.8.43.190:46251)
0925 15:19:24.705216 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:24.705224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:24.705255 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:19:24.705372 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:24.705379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:24.705427 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:24.705577 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:24.706267 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.707710 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.708771 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.709412 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.709540 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:24.712306 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:24.712310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:24.712311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:24.712322 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:24.714297 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:24.714698 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:24.714701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:24.714701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:24.714800 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:24.715001 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:24.715002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:24.715002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:24.715320 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:24.715469 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:24.715620 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 15:19:24.715771 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":55,"threads_started":1}
I20260925 15:19:24.719352 25429 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:24.720088 25429 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:24.727304 25429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41: Generated new TSK 0
I20260925 15:19:24.727480 25429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:19:24.755348 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46251
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:24.842626 25443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:24.842873 25443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:24.842926 25443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:24.844130 25443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:24.844204 25443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:24.844259 25443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:19:24.845755 25443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:24.846043 25443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:24.846276 25443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:24.851375 25449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:24.851352 25451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:24.851359 25448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:24.851779 25443 server_base.cc:1034] running on GCE node
I20260925 15:19:24.851990 25443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:24.852212 25443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:24.853348 25443 hybrid_clock.cc:648] HybridClock initialized: now 1790349564853322 us; error 42 us; skew 500 ppm
Sep 25 15:19:24 dist-test-slave-c5d6 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349564, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:24.855974 25443 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:19:24.856341 25443 webserver.cc:460] Webserver started at http://127.8.43.129:35809/ using document root <none> and password file <none>
I20260925 15:19:24.856501 25443 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:24.856537 25443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:24.856664 25443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:24.857659 25443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "52d82f9e10211fa4db4bec8a88299791"
server_key_iv: "fef418e68fa1e56b429b0b609c7c6e56"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.857952 25443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "52d82f9e10211fa4db4bec8a88299791"
server_key_iv: "fef418e68fa1e56b429b0b609c7c6e56"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.859326 25443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:19:24.860121 25458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:24.860383 25443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:24.860505 25443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "52d82f9e10211fa4db4bec8a88299791"
server_key_iv: "fef418e68fa1e56b429b0b609c7c6e56"
server_key_version: "encryptionkey@0"
I20260925 15:19:24.860597 25443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:24.869319 25443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:24.871599 25443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:24.871752 25443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:24.871981 25443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:24.872256 25443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:24.872308 25443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:24.872346 25443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:24.872372 25443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:24.878021 25443 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:42755
I20260925 15:19:24.878088 25571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:42755 every 8 connection(s)
I20260925 15:19:24.878418 25443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:24.880401 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 25443
I20260925 15:19:24.880537 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:24.880721 8366 external_mini_cluster.cc:1428] Setting key 78f205b43a0b358ef161c6a0a203bdbb
Sep 25 15:19:24 dist-test-slave-c5d6 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349564, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:24.889283 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:24.879279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:60255 (local address 127.8.43.190:46251)
0925 15:19:24.879340 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:24.879342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:24.879531 (+ 189us) server_negotiation.cc:407] Connection header received
0925 15:19:24.880066 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:24.880067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:24.880084 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:24.880138 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:24.881350 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.882094 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.883423 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.883632 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.883691 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:24.886174 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:24.886175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:24.886176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:24.886184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:24.887938 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:24.888310 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:24.888311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:24.888311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:24.888377 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:24.888604 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:24.888605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:24.888605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:24.888739 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:24.888811 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:24.889075 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:19:24.889169 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:19:24.889305 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:24.879175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46251 (local address 127.8.43.129:60255)
0925 15:19:24.879377 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:24.879428 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:24.879950 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:24.880205 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:24.880209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:24.880932 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:24.881224 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:24.881230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.882184 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.882185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:24.883323 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:24.883326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:24.883720 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:24.883721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:24.883863 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:24.884446 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:24.884454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:24.886075 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:24.888026 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:24.888033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:24.888035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:24.888234 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:24.888446 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:24.888447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:24.888447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:24.888540 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:24.888812 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:24.888816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:24.888923 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 15:19:24.889023 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:19:24.889139 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
I20260925 15:19:24.889846 25572 heartbeater.cc:347] Connected to a master server at 127.8.43.190:46251
I20260925 15:19:24.889971 25572 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:24.890146 25572 heartbeater.cc:511] Master 127.8.43.190:46251 requested a full tablet report, sending...
I20260925 15:19:24.890620 25381 ts_manager.cc:194] Registered new tserver with Master: 0ed3a9117f07439dac5c7bfa3dcb14c0 (127.8.43.129:42755)
I20260925 15:19:24.891324 25381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:60255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:19:24.906226 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46251
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:24.986708 25577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:24.986954 25577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:24.987002 25577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:24.988163 25577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:24.988235 25577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:24.988281 25577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:19:24.989713 25577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:24.989984 25577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:24.990214 25577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:24.995253 25585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:24.995287 25582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:24.995482 25577 server_base.cc:1034] running on GCE node
W20260925 15:19:24.995508 25583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:24.995774 25577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:24.996026 25577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:24.997210 25577 hybrid_clock.cc:648] HybridClock initialized: now 1790349564997194 us; error 45 us; skew 500 ppm
Sep 25 15:19:24 dist-test-slave-c5d6 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349564, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:24.999686 25577 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:19:25.000113 25577 webserver.cc:460] Webserver started at http://127.8.43.130:37487/ using document root <none> and password file <none>
I20260925 15:19:25.000340 25577 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:25.000406 25577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:25.000571 25577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:25.001602 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "80aa52b034c749bb8c0e5920cd2ec58f"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "81ca8a13d3650b262e473287a332b2ac"
server_key_iv: "6ebd9c53642f96f76d8842c29b740d81"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.002007 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "80aa52b034c749bb8c0e5920cd2ec58f"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "81ca8a13d3650b262e473287a332b2ac"
server_key_iv: "6ebd9c53642f96f76d8842c29b740d81"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.003487 25577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:25.004210 25592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.004383 25577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:25.004516 25577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "80aa52b034c749bb8c0e5920cd2ec58f"
format_stamp: "Formatted at 2026-09-25 15:19:24 on dist-test-slave-c5d6"
server_key: "81ca8a13d3650b262e473287a332b2ac"
server_key_iv: "6ebd9c53642f96f76d8842c29b740d81"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.004593 25577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:25.022987 25577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:25.024941 25577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:25.025110 25577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:25.025352 25577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:25.025679 25577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:25.025733 25577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.025775 25577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:25.025799 25577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.031665 25577 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:36151
I20260925 15:19:25.031769 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:36151 every 8 connection(s)
I20260925 15:19:25.032063 25577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:19:25 dist-test-slave-c5d6 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349564, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:25.040756 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.032837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:43171 (local address 127.8.43.190:46251)
0925 15:19:25.032920 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.032922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.033061 (+ 139us) server_negotiation.cc:407] Connection header received
0925 15:19:25.033551 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.033552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.033562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.033611 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:25.034536 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.035025 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.036055 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.036236 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.036290 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.038247 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:25.038248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:25.038248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:25.038255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:25.039671 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:25.040021 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:25.040022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:25.040022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:25.040069 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:25.040272 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:25.040272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:25.040272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:25.040364 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:25.040412 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.040629 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.040694 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:19:25.040971 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.032736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46251 (local address 127.8.43.130:43171)
0925 15:19:25.032922 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.032962 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.033466 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.033672 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.033675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.034256 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:25.034461 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.034467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.035101 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.035102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.035978 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.035980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.036308 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.036308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.036403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.036852 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:25.036859 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:25.038167 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:25.039736 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:25.039741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:25.039743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:25.039944 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:25.040126 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:25.040127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:25.040127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:25.040220 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:25.040415 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:25.040419 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:25.040528 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.040606 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.040715 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260925 15:19:25.041150 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 25577
I20260925 15:19:25.041260 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:25.041474 8366 external_mini_cluster.cc:1428] Setting key abe0a039f94f210c046d18ad89189886
I20260925 15:19:25.041497 25706 heartbeater.cc:347] Connected to a master server at 127.8.43.190:46251
I20260925 15:19:25.041584 25706 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:25.041778 25706 heartbeater.cc:511] Master 127.8.43.190:46251 requested a full tablet report, sending...
I20260925 15:19:25.042219 25381 ts_manager.cc:194] Registered new tserver with Master: 80aa52b034c749bb8c0e5920cd2ec58f (127.8.43.130:36151)
I20260925 15:19:25.042711 25381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:43171
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:19:25.063567 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46251
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:25.144874 25711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:25.145114 25711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:25.145149 25711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:25.146282 25711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:25.146350 25711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:25.146394 25711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:19:25.147894 25711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:25.148152 25711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:25.148375 25711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:25.153182 25716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.153198 25717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.153198 25719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:25.153434 25711 server_base.cc:1034] running on GCE node
I20260925 15:19:25.153638 25711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:25.153877 25711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:25.155027 25711 hybrid_clock.cc:648] HybridClock initialized: now 1790349565155009 us; error 34 us; skew 500 ppm
Sep 25 15:19:25 dist-test-slave-c5d6 krb5kdc[25336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349565, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:25.157398 25711 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:19:25.157797 25711 webserver.cc:460] Webserver started at http://127.8.43.131:36891/ using document root <none> and password file <none>
I20260925 15:19:25.157974 25711 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:25.158013 25711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:25.158099 25711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:25.159174 25711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "393688c20d69429096b8731a41d8bd9e"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "1a3a175a4c0f647597f3728fd0e17e91"
server_key_iv: "39553fe4d44be70f1824bc2b3575c597"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.159543 25711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "393688c20d69429096b8731a41d8bd9e"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "1a3a175a4c0f647597f3728fd0e17e91"
server_key_iv: "39553fe4d44be70f1824bc2b3575c597"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.160799 25711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:25.161495 25726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.161661 25711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:25.161742 25711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "393688c20d69429096b8731a41d8bd9e"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "1a3a175a4c0f647597f3728fd0e17e91"
server_key_iv: "39553fe4d44be70f1824bc2b3575c597"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.161794 25711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:25.180387 25711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:25.182240 25711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:25.182350 25711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:25.182541 25711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:25.182857 25711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:25.182891 25711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.182914 25711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:25.182961 25711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.188732 25711 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:36873
I20260925 15:19:25.188810 25839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:36873 every 8 connection(s)
I20260925 15:19:25.189110 25711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:19:25 dist-test-slave-c5d6 krb5kdc[25336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349565, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:25.197906 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.189829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:42221 (local address 127.8.43.190:46251)
0925 15:19:25.189890 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.189892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.190098 (+ 206us) server_negotiation.cc:407] Connection header received
0925 15:19:25.190712 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.190713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.190726 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.190766 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:25.191848 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.192345 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.193320 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.193520 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.193604 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.195631 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:25.195633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:25.195633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:25.195640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:25.196801 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:25.197167 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:25.197168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:25.197169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:25.197216 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:25.197415 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:25.197415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:25.197415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:25.197520 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:25.197568 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.197787 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.197845 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:19:25.198117 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.189814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:46251 (local address 127.8.43.131:42221)
0925 15:19:25.189979 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.190015 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.190600 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.190846 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.190855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.191473 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:25.191736 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.191743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.192443 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.192444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.193229 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.193231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.193611 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.193613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.193734 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.194223 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:25.194231 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:25.195548 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:25.196868 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:25.196873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:25.196874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:25.197103 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:25.197273 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:25.197273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:25.197273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:25.197363 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:25.197572 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:25.197576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:25.197680 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.197759 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.197868 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260925 15:19:25.198208 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 25711
I20260925 15:19:25.198330 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:25.198550 8366 external_mini_cluster.cc:1428] Setting key 30103d7066254e5fbdd958a5facb54bb
I20260925 15:19:25.198589 25840 heartbeater.cc:347] Connected to a master server at 127.8.43.190:46251
I20260925 15:19:25.198697 25840 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:25.198882 25840 heartbeater.cc:511] Master 127.8.43.190:46251 requested a full tablet report, sending...
I20260925 15:19:25.199385 25380 ts_manager.cc:194] Registered new tserver with Master: 393688c20d69429096b8731a41d8bd9e (127.8.43.131:36873)
I20260925 15:19:25.199755 25380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:42221
I20260925 15:19:25.210371 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:25.217497 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.211862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.8.43.190:46251)
0925 15:19:25.211943 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.211944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.212050 (+ 106us) server_negotiation.cc:407] Connection header received
0925 15:19:25.212097 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.212097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.212106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.212146 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:25.212650 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.213106 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.213796 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.213988 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.214044 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.215144 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:25.215145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:25.215145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:25.215151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:25.216565 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:25.216861 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:25.216861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:25.216862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:25.216907 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:25.217087 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:25.217088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:25.217088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:25.217211 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:25.217284 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.217378 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.217429 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:19:25.219166 25380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:25.220726 25380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:25.227043 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.223044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.8.43.131:36873)
0925 15:19:25.223569 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.223574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.223584 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:19:25.223623 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.223625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.223640 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.223885 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.223945 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.224596 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.225961 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.226591 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.226698 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.226763 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.226907 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.226951 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":97,"threads_started":1}
I20260925 15:19:25.227029 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.222948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:36873 (local address 127.0.0.1:52140)
0925 15:19:25.223184 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.223194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.223278 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.223740 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.223741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.223744 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.223807 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.223809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.224701 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.224702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.225857 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.225860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.226700 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.226701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.226789 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.226803 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.226883 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.226947 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":54,"threads_started":1}
I20260925 15:19:25.227463 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.223147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:42755 (local address 127.0.0.1:33966)
0925 15:19:25.223391 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.223401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.223442 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.223749 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.223750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.223753 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.223806 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.223807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.225152 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.225153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.226160 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.226162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.227083 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.227084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.227296 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.227301 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.227337 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.227374 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":37,"thread_start_us":36,"threads_started":1}
I20260925 15:19:25.227568 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.222931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:36151 (local address 127.0.0.1:45172)
0925 15:19:25.223120 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.223161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.223276 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.223638 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.223642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.223645 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.223720 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.223722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.225123 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.225125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.226199 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.226202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.227338 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.227339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.227400 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.227403 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.227444 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.227495 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260925 15:19:25.227870 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.223027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.8.43.130:36151)
0925 15:19:25.223348 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.223353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.223364 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:25.223411 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.223413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.223436 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.223540 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.223811 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.224991 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.226284 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.226976 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.227053 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.227110 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.227700 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.227757 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":70,"threads_started":1}
I20260925 15:19:25.228006 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.223352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.8.43.129:42755)
0925 15:19:25.223562 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.223566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.223578 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:19:25.223621 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.223623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.223638 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.223889 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.223978 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.225039 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.226243 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.226990 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.227754 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.227836 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.227861 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.227911 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
I20260925 15:19:25.227964 25774 tablet_service.cc:1467] Processing CreateTablet for tablet 5175034068034715abe1d645eb5a6f7a (DEFAULT_TABLE table=test-table [id=edaa0388e8424e2b964616265d64c638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:25.228356 25774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175034068034715abe1d645eb5a6f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:25.228488 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 5175034068034715abe1d645eb5a6f7a (DEFAULT_TABLE table=test-table [id=edaa0388e8424e2b964616265d64c638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:25.228627 25506 tablet_service.cc:1467] Processing CreateTablet for tablet 5175034068034715abe1d645eb5a6f7a (DEFAULT_TABLE table=test-table [id=edaa0388e8424e2b964616265d64c638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:25.228840 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175034068034715abe1d645eb5a6f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:25.228971 25506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5175034068034715abe1d645eb5a6f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:25.231307 25854 tablet_bootstrap.cc:492] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: Bootstrap starting.
I20260925 15:19:25.231880 25855 tablet_bootstrap.cc:492] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: Bootstrap starting.
I20260925 15:19:25.232116 25854 tablet_bootstrap.cc:654] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:25.232463 25856 tablet_bootstrap.cc:492] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0: Bootstrap starting.
I20260925 15:19:25.232602 25855 tablet_bootstrap.cc:654] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:25.233007 25854 log.cc:824] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:25.233032 25856 tablet_bootstrap.cc:654] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:25.233146 25855 log.cc:824] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:25.233668 25856 log.cc:824] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:25.233881 25854 tablet_bootstrap.cc:492] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: No bootstrap required, opened a new log
I20260925 15:19:25.233961 25854 ts_tablet_manager.cc:1397] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:19:25.234397 25856 tablet_bootstrap.cc:492] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0: No bootstrap required, opened a new log
I20260925 15:19:25.234454 25856 ts_tablet_manager.cc:1397] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:25.235584 25855 tablet_bootstrap.cc:492] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: No bootstrap required, opened a new log
I20260925 15:19:25.235690 25855 ts_tablet_manager.cc:1397] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:19:25.235879 25854 raft_consensus.cc:348] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.236071 25854 raft_consensus.cc:374] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:25.236107 25854 raft_consensus.cc:729] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80aa52b034c749bb8c0e5920cd2ec58f, State: Initialized, Role: FOLLOWER
I20260925 15:19:25.236218 25854 consensus_queue.cc:260] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.236505 25854 ts_tablet_manager.cc:1428] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:25.236426 25856 raft_consensus.cc:348] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.236624 25856 raft_consensus.cc:374] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:25.236645 25706 heartbeater.cc:503] Master 127.8.43.190:46251 was elected leader, sending a full tablet report...
I20260925 15:19:25.236672 25856 raft_consensus.cc:729] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed3a9117f07439dac5c7bfa3dcb14c0, State: Initialized, Role: FOLLOWER
I20260925 15:19:25.236788 25856 consensus_queue.cc:260] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.237089 25572 heartbeater.cc:503] Master 127.8.43.190:46251 was elected leader, sending a full tablet report...
I20260925 15:19:25.237089 25856 ts_tablet_manager.cc:1428] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:25.237782 25855 raft_consensus.cc:348] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.237979 25855 raft_consensus.cc:374] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:25.238049 25855 raft_consensus.cc:729] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 393688c20d69429096b8731a41d8bd9e, State: Initialized, Role: FOLLOWER
I20260925 15:19:25.238188 25855 consensus_queue.cc:260] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.238483 25855 ts_tablet_manager.cc:1428] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:25.238538 25840 heartbeater.cc:503] Master 127.8.43.190:46251 was elected leader, sending a full tablet report...
I20260925 15:19:25.248942 25860 raft_consensus.cc:482] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:25.249070 25860 raft_consensus.cc:504] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.249435 25860 leader_election.cc:288] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed3a9117f07439dac5c7bfa3dcb14c0 (127.8.43.129:42755), 393688c20d69429096b8731a41d8bd9e (127.8.43.131:36873)
I20260925 15:19:25.253331 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.249656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:37217 (local address 127.8.43.131:36873)
0925 15:19:25.249704 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.249706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.249841 (+ 135us) server_negotiation.cc:407] Connection header received
0925 15:19:25.249869 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.249870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.249880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.249939 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.250109 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.250861 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.252421 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.252992 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.253073 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.253083 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.253226 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.253268 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:19:25.253350 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.249595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:36873 (local address 127.8.43.130:37217)
0925 15:19:25.249757 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.249766 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.249816 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.249945 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.249945 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.249947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.250006 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.250007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.250938 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.250938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.252329 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.252333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.253080 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.253081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.253147 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.253150 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.253219 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.253261 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":155,"thread_start_us":91,"threads_started":1}
I20260925 15:19:25.253396 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.249644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:49783 (local address 127.8.43.129:42755)
0925 15:19:25.249685 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.249686 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.249696 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:19:25.249778 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.249779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.249791 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.249880 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.250109 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.250852 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.252421 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.253029 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.253090 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.253099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.253293 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.253327 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:19:25.253644 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.249589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:42755 (local address 127.8.43.130:49783)
0925 15:19:25.249650 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.249661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.249704 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.249889 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.249890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.249893 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.250006 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.250007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.250938 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.250938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.252329 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.252333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.253133 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.253134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.253205 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.253208 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.253276 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.253321 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":63}
I20260925 15:19:25.253793 25794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175034068034715abe1d645eb5a6f7a" candidate_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "393688c20d69429096b8731a41d8bd9e" is_pre_election: true
I20260925 15:19:25.253964 25794 raft_consensus.cc:2454] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80aa52b034c749bb8c0e5920cd2ec58f in term 0.
I20260925 15:19:25.253971 25526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175034068034715abe1d645eb5a6f7a" candidate_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" is_pre_election: true
I20260925 15:19:25.254138 25526 raft_consensus.cc:2454] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80aa52b034c749bb8c0e5920cd2ec58f in term 0.
I20260925 15:19:25.254201 25595 leader_election.cc:302] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 393688c20d69429096b8731a41d8bd9e, 80aa52b034c749bb8c0e5920cd2ec58f; no voters:
I20260925 15:19:25.254319 25860 raft_consensus.cc:2781] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:25.254390 25860 raft_consensus.cc:482] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:25.254419 25860 raft_consensus.cc:3037] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:25.255127 25860 raft_consensus.cc:504] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.255294 25860 leader_election.cc:288] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [CANDIDATE]: Term 1 election: Requested vote from peers 0ed3a9117f07439dac5c7bfa3dcb14c0 (127.8.43.129:42755), 393688c20d69429096b8731a41d8bd9e (127.8.43.131:36873)
I20260925 15:19:25.255424 25794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175034068034715abe1d645eb5a6f7a" candidate_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "393688c20d69429096b8731a41d8bd9e"
I20260925 15:19:25.255435 25526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5175034068034715abe1d645eb5a6f7a" candidate_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0"
I20260925 15:19:25.255504 25794 raft_consensus.cc:3037] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:25.255510 25526 raft_consensus.cc:3037] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:25.256184 25794 raft_consensus.cc:2454] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80aa52b034c749bb8c0e5920cd2ec58f in term 1.
I20260925 15:19:25.256290 25526 raft_consensus.cc:2454] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80aa52b034c749bb8c0e5920cd2ec58f in term 1.
I20260925 15:19:25.256345 25595 leader_election.cc:302] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 393688c20d69429096b8731a41d8bd9e, 80aa52b034c749bb8c0e5920cd2ec58f; no voters:
I20260925 15:19:25.256422 25860 raft_consensus.cc:2781] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:25.256603 25860 raft_consensus.cc:686] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 1 LEADER]: Becoming Leader. State: Replica: 80aa52b034c749bb8c0e5920cd2ec58f, State: Running, Role: LEADER
I20260925 15:19:25.256704 25860 consensus_queue.cc:237] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } }
I20260925 15:19:25.257503 25380 catalog_manager.cc:5495] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f reported cstate change: term changed from 0 to 1, leader changed from <none> to 80aa52b034c749bb8c0e5920cd2ec58f (127.8.43.130). New cstate: current_term: 1 leader_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80aa52b034c749bb8c0e5920cd2ec58f" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 } health_report { overall_health: UNKNOWN } } }
I20260925 15:19:25.266525 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.263998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.8.43.130:36151)
0925 15:19:25.264060 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.264062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.264081 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:19:25.264150 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.264151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.264167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.264270 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:25.264405 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.264976 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.265815 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.266010 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.266064 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.266195 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:25.266285 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:25.266345 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.266387 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.266404 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:19:25.268283 25526 raft_consensus.cc:1261] T 5175034068034715abe1d645eb5a6f7a P 0ed3a9117f07439dac5c7bfa3dcb14c0 [term 1 FOLLOWER]: Refusing update from remote peer 80aa52b034c749bb8c0e5920cd2ec58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:25.268328 25794 raft_consensus.cc:1261] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 1 FOLLOWER]: Refusing update from remote peer 80aa52b034c749bb8c0e5920cd2ec58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:25.268644 25864 consensus_queue.cc:1035] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "393688c20d69429096b8731a41d8bd9e" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:25.268757 25860 consensus_queue.cc:1035] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed3a9117f07439dac5c7bfa3dcb14c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:25.271951 25868 mvcc.cc:205] Tried to move back new op lower bound from 7333271819336282112 to 7333271819292139520. Current Snapshot: MvccSnapshot[applied={T|T < 7333271819336282112}]
I20260925 15:19:25.273351 25381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:25.273443 25381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:25.274186 25381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83302fcdfaf248b7b2d7049ce5d84a41: Sending DeleteTablet for 3 replicas of tablet 5175034068034715abe1d645eb5a6f7a
I20260925 15:19:25.274494 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175034068034715abe1d645eb5a6f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:25 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:45172
I20260925 15:19:25.274531 25774 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175034068034715abe1d645eb5a6f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:25 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:52140
I20260925 15:19:25.274535 25506 tablet_service.cc:1514] Processing DeleteTablet for tablet 5175034068034715abe1d645eb5a6f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:25 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:33966
I20260925 15:19:25.274780 25877 tablet_replica.cc:330] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: stopping tablet replica
I20260925 15:19:25.274879 25877 raft_consensus.cc:2229] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:25.274989 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 25443
I20260925 15:19:25.275285 25877 raft_consensus.cc:2258] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:25.275876 25877 ts_tablet_manager.cc:1905] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:25.277329 25878 tablet_replica.cc:330] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: stopping tablet replica
I20260925 15:19:25.277459 25878 raft_consensus.cc:2229] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:25.277619 25878 raft_consensus.cc:2258] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:25.277608 25877 ts_tablet_manager.cc:1918] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:25.277678 25877 log.cc:1196] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/5175034068034715abe1d645eb5a6f7a
I20260925 15:19:25.278059 25877 ts_tablet_manager.cc:1939] T 5175034068034715abe1d645eb5a6f7a P 80aa52b034c749bb8c0e5920cd2ec58f: Deleting consensus metadata
I20260925 15:19:25.278467 25365 catalog_manager.cc:4841] TS 80aa52b034c749bb8c0e5920cd2ec58f (127.8.43.130:36151): tablet 5175034068034715abe1d645eb5a6f7a (table test-table [id=edaa0388e8424e2b964616265d64c638]) successfully deleted
I20260925 15:19:25.278712 25878 ts_tablet_manager.cc:1905] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:25.280318 25878 ts_tablet_manager.cc:1918] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:25.280395 25878 log.cc:1196] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/5175034068034715abe1d645eb5a6f7a
I20260925 15:19:25.280675 25878 ts_tablet_manager.cc:1939] T 5175034068034715abe1d645eb5a6f7a P 393688c20d69429096b8731a41d8bd9e: Deleting consensus metadata
I20260925 15:19:25.281289 25367 catalog_manager.cc:4841] TS 393688c20d69429096b8731a41d8bd9e (127.8.43.131:36873): tablet 5175034068034715abe1d645eb5a6f7a (table test-table [id=edaa0388e8424e2b964616265d64c638]) successfully deleted
W20260925 15:19:25.281344 25365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:42755 (error 108)
I20260925 15:19:25.281579 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 25577
I20260925 15:19:25.282173 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.281695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:42755 (local address 127.0.0.1:33976)
0925 15:19:25.282092 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.282131 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
W20260925 15:19:25.282357 25365 catalog_manager.cc:4584] TS 0ed3a9117f07439dac5c7bfa3dcb14c0 (127.8.43.129:42755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5175034068034715abe1d645eb5a6f7a: Network error: Client connection negotiation failed: client connection to 127.8.43.129:42755: connect: Connection refused (error 111)
I20260925 15:19:25.286069 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 25711
I20260925 15:19:25.290529 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 25349
2026-09-25T15:19:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1889 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:25Z Disabled control of system clock
I20260925 15:19:25.318020 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44575
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:42279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:19:25.397692 25884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:25.397940 25884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:25.397974 25884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:25.399139 25884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:25.399204 25884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:25.399233 25884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:25.399295 25884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:25.399312 25884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:25.400661 25884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:25.400911 25884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:25.401207 25884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:25.405970 25892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.405975 25890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.406018 25889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:25.406229 25884 server_base.cc:1034] running on GCE node
I20260925 15:19:25.406519 25884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:25.406803 25884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:25.407970 25884 hybrid_clock.cc:648] HybridClock initialized: now 1790349565407934 us; error 68 us; skew 500 ppm
I20260925 15:19:25.409194 25884 webserver.cc:460] Webserver started at http://127.8.43.190:43577/ using document root <none> and password file <none>
I20260925 15:19:25.409478 25884 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:25.409554 25884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:25.409701 25884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:25.410778 25884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "ef4dd73732b64e94a76846f373440d09"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "ab417b00834f3409e5c9cbfd1d3dd475"
server_key_iv: "e248e7e195be325db79770332244d94a"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.411145 25884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "ef4dd73732b64e94a76846f373440d09"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "ab417b00834f3409e5c9cbfd1d3dd475"
server_key_iv: "e248e7e195be325db79770332244d94a"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.412555 25884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:25.413300 25898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.413473 25884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:25.413578 25884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "ef4dd73732b64e94a76846f373440d09"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "ab417b00834f3409e5c9cbfd1d3dd475"
server_key_iv: "e248e7e195be325db79770332244d94a"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.413651 25884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:25.433578 25884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:25.433920 25884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:25.434087 25884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:25.437884 25884 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44575
I20260925 15:19:25.437923 25950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44575 every 8 connection(s)
I20260925 15:19:25.438298 25884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:25.439307 25951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:25.441496 25951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09: Bootstrap starting.
I20260925 15:19:25.442059 25951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:25.442116 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 25884
I20260925 15:19:25.442220 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:25.442409 8366 external_mini_cluster.cc:1428] Setting key 816b512aa9651e23cfe3e1d73717fe5f
I20260925 15:19:25.442417 25951 log.cc:824] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:25.443372 25951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09: No bootstrap required, opened a new log
I20260925 15:19:25.445152 25951 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4dd73732b64e94a76846f373440d09" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44575 } }
I20260925 15:19:25.445308 25951 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:25.445355 25951 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4dd73732b64e94a76846f373440d09, State: Initialized, Role: FOLLOWER
I20260925 15:19:25.445492 25951 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4dd73732b64e94a76846f373440d09" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44575 } }
I20260925 15:19:25.445571 25951 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:25.445614 25951 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:25.445662 25951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:25.446173 25951 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4dd73732b64e94a76846f373440d09" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44575 } }
I20260925 15:19:25.446293 25951 leader_election.cc:302] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef4dd73732b64e94a76846f373440d09; no voters:
I20260925 15:19:25.446503 25951 leader_election.cc:288] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:25.446529 25956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:25.446718 25956 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [term 1 LEADER]: Becoming Leader. State: Replica: ef4dd73732b64e94a76846f373440d09, State: Running, Role: LEADER
I20260925 15:19:25.446825 25951 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:25.446847 25956 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4dd73732b64e94a76846f373440d09" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44575 } }
I20260925 15:19:25.447252 25956 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4dd73732b64e94a76846f373440d09. Latest consensus state: current_term: 1 leader_uuid: "ef4dd73732b64e94a76846f373440d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4dd73732b64e94a76846f373440d09" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44575 } } }
I20260925 15:19:25.447341 25956 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:25.448036 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.442893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.8.43.190:44575)
0925 15:19:25.443075 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.443084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.443117 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:25.443640 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.443647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.443706 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.443845 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:25.444088 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.444944 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.445944 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.446414 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.446545 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.447287 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:25.447291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:25.447291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:25.447525 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:25.447564 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:25.447580 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:25.447647 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.447761 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.447843 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260925 15:19:25.448088 25963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:25.447999 25958 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef4dd73732b64e94a76846f373440d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4dd73732b64e94a76846f373440d09" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44575 } } }
I20260925 15:19:25.448897 25958 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:25.448469 25963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:25.450258 25963 catalog_manager.cc:1347] Generated new cluster ID: b32f7490690e4b7f8370cfd4e29db934
I20260925 15:19:25.450305 25963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:25.455093 25963 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:25.455768 25963 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:25.465168 25963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09: Generated new TSK 0
I20260925 15:19:25.465353 25963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:25.473289 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44575
--builtin_ntp_servers=127.8.43.148:42279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:19:25.570477 25975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:25.570665 25975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:25.570686 25975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:25.571830 25975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:25.571867 25975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:25.571892 25975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:19:25.573181 25975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:25.573418 25975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:25.573704 25975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:25.578553 25981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.578564 25980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.578572 25983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:25.578912 25975 server_base.cc:1034] running on GCE node
I20260925 15:19:25.579146 25975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:25.579442 25975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:25.580646 25975 hybrid_clock.cc:648] HybridClock initialized: now 1790349565580629 us; error 42 us; skew 500 ppm
I20260925 15:19:25.582098 25975 webserver.cc:460] Webserver started at http://127.8.43.129:36403/ using document root <none> and password file <none>
I20260925 15:19:25.582370 25975 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:25.582430 25975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:25.582566 25975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:25.583897 25975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "240fa60c5f16431b9c0e39a77af00c3c"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "5bf3be89628b800311cf5d7fd6b84fb6"
server_key_iv: "ff444bc0e06edc619f86abfdf65a0960"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.584264 25975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "240fa60c5f16431b9c0e39a77af00c3c"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "5bf3be89628b800311cf5d7fd6b84fb6"
server_key_iv: "ff444bc0e06edc619f86abfdf65a0960"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.585752 25975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:25.586547 25989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.586724 25975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:25.586818 25975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "240fa60c5f16431b9c0e39a77af00c3c"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "5bf3be89628b800311cf5d7fd6b84fb6"
server_key_iv: "ff444bc0e06edc619f86abfdf65a0960"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.586891 25975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:25.601182 25975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:25.601478 25975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:25.601624 25975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:25.601835 25975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:25.602161 25975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:25.602196 25975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.602217 25975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:25.602262 25975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.607614 25975 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:46039
I20260925 15:19:25.607705 26102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:46039 every 8 connection(s)
I20260925 15:19:25.608009 25975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:25.608552 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 25975
I20260925 15:19:25.608665 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:25.608881 8366 external_mini_cluster.cc:1428] Setting key 71d994a348a1aa293be57755fc92659c
I20260925 15:19:25.609937 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44575
--builtin_ntp_servers=127.8.43.148:42279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:19:25.613926 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.608728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:34447 (local address 127.8.43.190:44575)
0925 15:19:25.608779 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.608781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.609109 (+ 328us) server_negotiation.cc:407] Connection header received
0925 15:19:25.609769 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.609770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.609783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.609851 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:25.610505 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.611302 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.612299 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.612661 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.612745 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.613626 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:25.613627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:25.613627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:25.613632 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:25.613649 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:25.613654 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:25.613723 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.613784 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.613812 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:19:25.614040 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.608737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44575 (local address 127.8.43.129:34447)
0925 15:19:25.608950 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.608998 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.609663 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.609907 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.609910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.609918 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:25.610409 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.610413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.611418 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.611419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.612210 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.612212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.612755 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.612756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.612872 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.613447 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:25.613448 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:25.613481 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:25.613501 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:25.613724 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:25.613728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:25.613730 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.613779 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.613853 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":114,"threads_started":1}
I20260925 15:19:25.614486 26103 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44575
I20260925 15:19:25.614588 26103 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:25.614754 26103 heartbeater.cc:511] Master 127.8.43.190:44575 requested a full tablet report, sending...
I20260925 15:19:25.615334 25915 ts_manager.cc:194] Registered new tserver with Master: 240fa60c5f16431b9c0e39a77af00c3c (127.8.43.129:46039)
I20260925 15:19:25.616304 25915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.129:34447
W20260925 15:19:25.691645 26106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:25.691843 26106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:25.691865 26106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:25.692921 26106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:25.692951 26106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:25.692975 26106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:19:25.694212 26106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:25.694392 26106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:25.694630 26106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:25.699467 26111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.699467 26112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:25.699640 26106 server_base.cc:1034] running on GCE node
W20260925 15:19:25.699486 26114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:25.699965 26106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:25.700222 26106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:25.701380 26106 hybrid_clock.cc:648] HybridClock initialized: now 1790349565701369 us; error 30 us; skew 500 ppm
I20260925 15:19:25.702652 26106 webserver.cc:460] Webserver started at http://127.8.43.130:42467/ using document root <none> and password file <none>
I20260925 15:19:25.702895 26106 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:25.702966 26106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:25.703140 26106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:25.704239 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "4d4657006f114292b5c3c000d674e789"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "7ab2e500d66d2bdb0289b6ae3e030688"
server_key_iv: "3438729e9cc467a98c0b8d7a59862d3c"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.704603 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "4d4657006f114292b5c3c000d674e789"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "7ab2e500d66d2bdb0289b6ae3e030688"
server_key_iv: "3438729e9cc467a98c0b8d7a59862d3c"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.705952 26106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:25.706669 26120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.706843 26106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:25.706941 26106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "4d4657006f114292b5c3c000d674e789"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "7ab2e500d66d2bdb0289b6ae3e030688"
server_key_iv: "3438729e9cc467a98c0b8d7a59862d3c"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.707034 26106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:25.729573 26106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:25.729907 26106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:25.730068 26106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:25.730293 26106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:25.730631 26106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:25.730686 26106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.730728 26106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:25.730757 26106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.736562 26106 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:39613
I20260925 15:19:25.736613 26233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:39613 every 8 connection(s)
I20260925 15:19:25.736932 26106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:25.742357 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.737698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:43917 (local address 127.8.43.190:44575)
0925 15:19:25.737756 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.737758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.738066 (+ 308us) server_negotiation.cc:407] Connection header received
0925 15:19:25.738624 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.738625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.738637 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.738709 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:25.739454 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.739962 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.740970 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.741164 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.741251 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.742061 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:25.742062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:25.742063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:25.742067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:25.742090 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:25.742094 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:25.742155 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.742249 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.742276 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:19:25.742619 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.737682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44575 (local address 127.8.43.130:43917)
0925 15:19:25.737889 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.737960 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.738511 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.738791 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.738795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.738802 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:25.739353 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.739357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.740068 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.740069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.740859 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.740861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.741260 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.741261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.741392 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.741899 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:25.741899 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:25.741936 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:25.741961 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:25.742155 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:25.742161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:25.742165 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.742251 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.742353 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260925 15:19:25.743180 26234 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44575
I20260925 15:19:25.743304 26234 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:25.743453 26234 heartbeater.cc:511] Master 127.8.43.190:44575 requested a full tablet report, sending...
I20260925 15:19:25.743836 25915 ts_manager.cc:194] Registered new tserver with Master: 4d4657006f114292b5c3c000d674e789 (127.8.43.130:39613)
I20260925 15:19:25.744213 25915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.130:43917
I20260925 15:19:25.745559 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 26106
I20260925 15:19:25.745632 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:25.745774 8366 external_mini_cluster.cc:1428] Setting key 5098cf2afc4701f128a39c8414292ca2
I20260925 15:19:25.746599 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44575
--builtin_ntp_servers=127.8.43.148:42279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:19:25.826983 26237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:25.827229 26237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:25.827297 26237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:25.828498 26237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:25.828568 26237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:25.828612 26237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:19:25.829960 26237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:25.830217 26237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:25.830510 26237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:25.835309 26243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.835346 26242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:25.835316 26245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:25.835610 26237 server_base.cc:1034] running on GCE node
I20260925 15:19:25.835819 26237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:25.836086 26237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:25.837241 26237 hybrid_clock.cc:648] HybridClock initialized: now 1790349565837233 us; error 28 us; skew 500 ppm
I20260925 15:19:25.838402 26237 webserver.cc:460] Webserver started at http://127.8.43.131:42271/ using document root <none> and password file <none>
I20260925 15:19:25.838672 26237 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:25.838748 26237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:25.838912 26237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:25.840051 26237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "c1cc025f3ad94f32ad49be4626cc4b34"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "eee41ad06be20060c36f3a42fc2886af"
server_key_iv: "0aa61a821564fd83cc1213f7c576f3ed"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.840422 26237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "c1cc025f3ad94f32ad49be4626cc4b34"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "eee41ad06be20060c36f3a42fc2886af"
server_key_iv: "0aa61a821564fd83cc1213f7c576f3ed"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.841912 26237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:19:25.842763 26251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.843034 26237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:25.843138 26237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "c1cc025f3ad94f32ad49be4626cc4b34"
format_stamp: "Formatted at 2026-09-25 15:19:25 on dist-test-slave-c5d6"
server_key: "eee41ad06be20060c36f3a42fc2886af"
server_key_iv: "0aa61a821564fd83cc1213f7c576f3ed"
server_key_version: "encryptionkey@0"
I20260925 15:19:25.843252 26237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:25.851452 26237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:25.851752 26237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:25.851907 26237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:25.852145 26237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:25.852473 26237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:25.852527 26237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.852561 26237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:25.852589 26237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:25.858044 26237 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:35919
I20260925 15:19:25.858141 26364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:35919 every 8 connection(s)
I20260925 15:19:25.858492 26237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:25.861294 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 26237
I20260925 15:19:25.861397 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:25.861588 8366 external_mini_cluster.cc:1428] Setting key c4ce30fa41c82a4ae9451068d602ac85
I20260925 15:19:25.863633 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.859304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:54933 (local address 127.8.43.190:44575)
0925 15:19:25.859368 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.859371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.859587 (+ 216us) server_negotiation.cc:407] Connection header received
0925 15:19:25.860071 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.860072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.860082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.860136 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:25.860750 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.861342 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.862372 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.862611 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.862663 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.863435 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:25.863436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:25.863437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:25.863440 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:25.863453 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:25.863457 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:25.863520 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.863557 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.863577 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:19:25.863859 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.859293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44575 (local address 127.8.43.131:54933)
0925 15:19:25.859464 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.859499 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.859962 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.860189 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.860192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.860199 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:25.860683 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.860687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.861454 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.861454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.862292 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.862295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.862681 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.862681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.862788 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.863313 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:25.863314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:25.863340 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:25.863362 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:25.863506 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:25.863508 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:25.863510 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.863552 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.863623 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260925 15:19:25.864332 26365 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44575
I20260925 15:19:25.864408 26365 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:25.864540 26365 heartbeater.cc:511] Master 127.8.43.190:44575 requested a full tablet report, sending...
I20260925 15:19:25.864841 25915 ts_manager.cc:194] Registered new tserver with Master: c1cc025f3ad94f32ad49be4626cc4b34 (127.8.43.131:35919)
I20260925 15:19:25.865234 25915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.131:54933
I20260925 15:19:25.873543 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:25.879083 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.874910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.8.43.190:44575)
0925 15:19:25.874980 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.874982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.875123 (+ 141us) server_negotiation.cc:407] Connection header received
0925 15:19:25.875189 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.875190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.875202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.875277 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:25.875506 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.876338 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.877613 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.877960 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.878048 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.878775 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:25.878776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:25.878776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:25.878800 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:25.878816 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:25.878820 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:25.878880 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.878950 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.878975 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:19:25.880847 25915 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:25.883013 25915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:25.894923 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.891171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.8.43.131:35919)
0925 15:19:25.891364 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.891369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.891380 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:25.891420 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.891422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.891437 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.891555 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.891727 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.892754 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.893853 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.894529 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.894615 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.894707 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.894794 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.894830 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260925 15:19:25.894912 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.890837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35919 (local address 127.0.0.1:33156)
0925 15:19:25.890951 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.891013 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.891080 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.891563 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.891566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.891568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.891640 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.891642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.892835 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.892836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.893760 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.893763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.894615 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.894616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.894692 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.894706 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.894780 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.894833 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":46,"thread_start_us":60,"threads_started":1}
I20260925 15:19:25.895370 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.891044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:39613 (local address 127.0.0.1:43376)
0925 15:19:25.891572 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.891582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.891641 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.891825 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.891826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.891828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.891885 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.891886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.892899 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.892900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.894116 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.894119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.895105 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.895106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.895165 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.895168 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.895213 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.895281 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":40,"thread_start_us":81,"threads_started":1}
I20260925 15:19:25.895604 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.891376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.8.43.130:39613)
0925 15:19:25.891556 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.891560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.891627 (+ 67us) server_negotiation.cc:407] Connection header received
0925 15:19:25.891716 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.891719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.891733 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.891830 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.891938 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.892808 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.894203 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.894913 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.894980 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.895072 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.895483 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.895529 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260925 15:19:25.895983 26299 tablet_service.cc:1467] Processing CreateTablet for tablet 1984bca0f9934ae593f20be0b73974d1 (DEFAULT_TABLE table=test-table [id=b1f2d3695d354c9d91c8115ab92f4d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:25.896205 26168 tablet_service.cc:1467] Processing CreateTablet for tablet 1984bca0f9934ae593f20be0b73974d1 (DEFAULT_TABLE table=test-table [id=b1f2d3695d354c9d91c8115ab92f4d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:25.896376 26299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1984bca0f9934ae593f20be0b73974d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:25.896552 26168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1984bca0f9934ae593f20be0b73974d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:25.896598 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.890849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46039 (local address 127.0.0.1:39420)
0925 15:19:25.891304 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.891314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.891357 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.892858 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.892859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.892861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.892900 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.892901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.894178 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.894179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.895281 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.895283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.896357 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.896359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.896421 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.896425 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.896491 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.896539 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":94,"threads_started":1}
I20260925 15:19:25.896831 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.891146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.8.43.129:46039)
0925 15:19:25.891684 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.891689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.891699 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:19:25.891736 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.891738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.891753 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.891824 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.892961 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.894081 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.895344 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.896151 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.896221 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.896300 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.896694 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.896745 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":134,"threads_started":1}
I20260925 15:19:25.897578 26037 tablet_service.cc:1467] Processing CreateTablet for tablet 1984bca0f9934ae593f20be0b73974d1 (DEFAULT_TABLE table=test-table [id=b1f2d3695d354c9d91c8115ab92f4d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:25.897891 26037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1984bca0f9934ae593f20be0b73974d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:25.899603 26379 tablet_bootstrap.cc:492] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Bootstrap starting.
I20260925 15:19:25.899757 26380 tablet_bootstrap.cc:492] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: Bootstrap starting.
I20260925 15:19:25.900360 26379 tablet_bootstrap.cc:654] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:25.900367 26380 tablet_bootstrap.cc:654] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:25.900701 26379 log.cc:824] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:25.900702 26380 log.cc:824] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:25.900957 26381 tablet_bootstrap.cc:492] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c: Bootstrap starting.
I20260925 15:19:25.901547 26381 tablet_bootstrap.cc:654] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:25.901571 26379 tablet_bootstrap.cc:492] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: No bootstrap required, opened a new log
I20260925 15:19:25.901618 26379 ts_tablet_manager.cc:1397] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:25.901787 26380 tablet_bootstrap.cc:492] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: No bootstrap required, opened a new log
I20260925 15:19:25.901859 26380 ts_tablet_manager.cc:1397] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:25.901925 26381 log.cc:824] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:25.902788 26381 tablet_bootstrap.cc:492] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c: No bootstrap required, opened a new log
I20260925 15:19:25.902902 26381 ts_tablet_manager.cc:1397] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:25.903944 26379 raft_consensus.cc:348] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.904103 26379 raft_consensus.cc:374] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:25.904150 26379 raft_consensus.cc:729] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4657006f114292b5c3c000d674e789, State: Initialized, Role: FOLLOWER
I20260925 15:19:25.904294 26379 consensus_queue.cc:260] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.904593 26234 heartbeater.cc:503] Master 127.8.43.190:44575 was elected leader, sending a full tablet report...
I20260925 15:19:25.904611 26380 raft_consensus.cc:348] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.904778 26380 raft_consensus.cc:374] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:25.904843 26380 raft_consensus.cc:729] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cc025f3ad94f32ad49be4626cc4b34, State: Initialized, Role: FOLLOWER
I20260925 15:19:25.904582 26379 ts_tablet_manager.cc:1428] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:25.904981 26380 consensus_queue.cc:260] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.905228 26380 ts_tablet_manager.cc:1428] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 15:19:25.905311 26365 heartbeater.cc:503] Master 127.8.43.190:44575 was elected leader, sending a full tablet report...
I20260925 15:19:25.905530 26381 raft_consensus.cc:348] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.905699 26381 raft_consensus.cc:374] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:25.905768 26381 raft_consensus.cc:729] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240fa60c5f16431b9c0e39a77af00c3c, State: Initialized, Role: FOLLOWER
I20260925 15:19:25.905900 26381 consensus_queue.cc:260] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.906201 26381 ts_tablet_manager.cc:1428] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:25.906237 26103 heartbeater.cc:503] Master 127.8.43.190:44575 was elected leader, sending a full tablet report...
I20260925 15:19:25.955471 26386 raft_consensus.cc:482] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:25.955607 26386 raft_consensus.cc:504] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.956036 26386 leader_election.cc:288] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 240fa60c5f16431b9c0e39a77af00c3c (127.8.43.129:46039), 4d4657006f114292b5c3c000d674e789 (127.8.43.130:39613)
I20260925 15:19:25.959136 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.956390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:47243 (local address 127.8.43.130:39613)
0925 15:19:25.956437 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.956439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.956603 (+ 164us) server_negotiation.cc:407] Connection header received
0925 15:19:25.956678 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.956679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.956694 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.956766 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.956916 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.957403 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.958396 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.958780 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.958876 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.958884 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.959031 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.959076 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:19:25.959142 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.956328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:39613 (local address 127.8.43.131:47243)
0925 15:19:25.956515 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.956523 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.956611 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.956768 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.956768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.956770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.956845 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.956846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.957497 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.957498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.958301 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.958303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.958876 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.958877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.958950 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.958953 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.959014 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.959054 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":117,"threads_started":1}
I20260925 15:19:25.959203 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.956431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:51419 (local address 127.8.43.129:46039)
0925 15:19:25.956481 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.956483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.956494 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:25.956549 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.956550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.956570 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.956649 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:25.956830 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.957388 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.958395 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.958797 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.958926 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.958935 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.959093 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.959128 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:19:25.959259 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.956216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46039 (local address 127.8.43.131:51419)
0925 15:19:25.956377 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:25.956390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:25.956475 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:25.956656 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:25.956657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:25.956659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:25.956752 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.956754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.957502 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.957503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.958292 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:25.958294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.958925 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.958926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:25.959000 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.959003 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:25.959085 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:19:25.959128 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260925 15:19:25.959626 26188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1984bca0f9934ae593f20be0b73974d1" candidate_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4657006f114292b5c3c000d674e789" is_pre_election: true
I20260925 15:19:25.959786 26188 raft_consensus.cc:2454] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1cc025f3ad94f32ad49be4626cc4b34 in term 0.
I20260925 15:19:25.959867 26057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1984bca0f9934ae593f20be0b73974d1" candidate_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240fa60c5f16431b9c0e39a77af00c3c" is_pre_election: true
I20260925 15:19:25.960028 26057 raft_consensus.cc:2454] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1cc025f3ad94f32ad49be4626cc4b34 in term 0.
I20260925 15:19:25.960067 26254 leader_election.cc:302] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4657006f114292b5c3c000d674e789, c1cc025f3ad94f32ad49be4626cc4b34; no voters:
I20260925 15:19:25.960191 26386 raft_consensus.cc:2781] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:25.960296 26386 raft_consensus.cc:482] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:25.960338 26386 raft_consensus.cc:3037] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:25.961125 26386 raft_consensus.cc:504] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.961280 26386 leader_election.cc:288] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [CANDIDATE]: Term 1 election: Requested vote from peers 240fa60c5f16431b9c0e39a77af00c3c (127.8.43.129:46039), 4d4657006f114292b5c3c000d674e789 (127.8.43.130:39613)
I20260925 15:19:25.961427 26057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1984bca0f9934ae593f20be0b73974d1" candidate_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240fa60c5f16431b9c0e39a77af00c3c"
I20260925 15:19:25.961464 26188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1984bca0f9934ae593f20be0b73974d1" candidate_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4657006f114292b5c3c000d674e789"
I20260925 15:19:25.961521 26057 raft_consensus.cc:3037] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:25.961566 26188 raft_consensus.cc:3037] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:25.962303 26057 raft_consensus.cc:2454] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1cc025f3ad94f32ad49be4626cc4b34 in term 1.
I20260925 15:19:25.962360 26188 raft_consensus.cc:2454] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1cc025f3ad94f32ad49be4626cc4b34 in term 1.
I20260925 15:19:25.962508 26254 leader_election.cc:302] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240fa60c5f16431b9c0e39a77af00c3c, c1cc025f3ad94f32ad49be4626cc4b34; no voters:
I20260925 15:19:25.962595 26386 raft_consensus.cc:2781] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:25.962754 26386 raft_consensus.cc:686] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 1 LEADER]: Becoming Leader. State: Replica: c1cc025f3ad94f32ad49be4626cc4b34, State: Running, Role: LEADER
I20260925 15:19:25.962977 26386 consensus_queue.cc:237] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } }
I20260925 15:19:25.963871 25915 catalog_manager.cc:5495] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1cc025f3ad94f32ad49be4626cc4b34 (127.8.43.131). New cstate: current_term: 1 leader_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1cc025f3ad94f32ad49be4626cc4b34" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 } health_report { overall_health: UNKNOWN } } }
I20260925 15:19:25.987254 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:25.984907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.8.43.131:35919)
0925 15:19:25.984968 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:25.984970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:25.984978 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:19:25.985069 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:25.985070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:25.985086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:25.985148 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:25.985258 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.985705 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.986411 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:25.986589 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:25.986648 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:25.986847 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:25.986940 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:25.987000 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:25.987057 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:19:25.987092 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 15:19:25.987541 26235 tablet.cc:2369] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:25.989147 26188 raft_consensus.cc:1261] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 1 FOLLOWER]: Refusing update from remote peer c1cc025f3ad94f32ad49be4626cc4b34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:25.989287 26057 raft_consensus.cc:1261] T 1984bca0f9934ae593f20be0b73974d1 P 240fa60c5f16431b9c0e39a77af00c3c [term 1 FOLLOWER]: Refusing update from remote peer c1cc025f3ad94f32ad49be4626cc4b34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:25.989517 26386 consensus_queue.cc:1035] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4657006f114292b5c3c000d674e789" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:25.989692 26389 consensus_queue.cc:1035] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "240fa60c5f16431b9c0e39a77af00c3c" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:25.992758 26393 mvcc.cc:205] Tried to move back new op lower bound from 7333271822289002496 to 7333271822185185280. Current Snapshot: MvccSnapshot[applied={T|T < 7333271822289002496 or (T in {7333271822289002496})}]
I20260925 15:19:25.992755 26394 mvcc.cc:205] Tried to move back new op lower bound from 7333271822289002496 to 7333271822185185280. Current Snapshot: MvccSnapshot[applied={T|T < 7333271822289002496}]
I20260925 15:19:25.994134 25914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:25.994226 25914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:25.995002 25914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef4dd73732b64e94a76846f373440d09: Sending DeleteTablet for 3 replicas of tablet 1984bca0f9934ae593f20be0b73974d1
I20260925 15:19:25.995316 26299 tablet_service.cc:1514] Processing DeleteTablet for tablet 1984bca0f9934ae593f20be0b73974d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:25 UTC) from {username='slave'} at 127.0.0.1:33156
I20260925 15:19:25.995394 26037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1984bca0f9934ae593f20be0b73974d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:25 UTC) from {username='slave'} at 127.0.0.1:39420
I20260925 15:19:25.995394 26168 tablet_service.cc:1514] Processing DeleteTablet for tablet 1984bca0f9934ae593f20be0b73974d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:25 UTC) from {username='slave'} at 127.0.0.1:43376
I20260925 15:19:25.995643 26401 tablet_replica.cc:330] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: stopping tablet replica
I20260925 15:19:25.995779 26401 raft_consensus.cc:2229] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:25.995864 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 25975
I20260925 15:19:25.995954 26401 raft_consensus.cc:2258] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:25.997763 26401 ts_tablet_manager.cc:1905] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:25.999351 26403 tablet_replica.cc:330] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: stopping tablet replica
I20260925 15:19:25.999516 26403 raft_consensus.cc:2229] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:25.999693 26403 raft_consensus.cc:2258] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:26.000005 26401 ts_tablet_manager.cc:1918] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:26.000128 26401 log.cc:1196] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/1984bca0f9934ae593f20be0b73974d1
I20260925 15:19:26.000211 26403 ts_tablet_manager.cc:1905] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:26.000442 26401 ts_tablet_manager.cc:1939] T 1984bca0f9934ae593f20be0b73974d1 P c1cc025f3ad94f32ad49be4626cc4b34: Deleting consensus metadata
I20260925 15:19:26.000891 25900 catalog_manager.cc:4841] TS c1cc025f3ad94f32ad49be4626cc4b34 (127.8.43.131:35919): tablet 1984bca0f9934ae593f20be0b73974d1 (table test-table [id=b1f2d3695d354c9d91c8115ab92f4d7a]) successfully deleted
I20260925 15:19:26.001876 26403 ts_tablet_manager.cc:1918] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:26.001950 26403 log.cc:1196] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/1984bca0f9934ae593f20be0b73974d1
I20260925 15:19:26.002208 26403 ts_tablet_manager.cc:1939] T 1984bca0f9934ae593f20be0b73974d1 P 4d4657006f114292b5c3c000d674e789: Deleting consensus metadata
W20260925 15:19:26.002537 25901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:46039 (error 108)
I20260925 15:19:26.002772 25901 catalog_manager.cc:4841] TS 4d4657006f114292b5c3c000d674e789 (127.8.43.130:39613): tablet 1984bca0f9934ae593f20be0b73974d1 (table test-table [id=b1f2d3695d354c9d91c8115ab92f4d7a]) successfully deleted
W20260925 15:19:26.003103 26374 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:26.002935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46039 (local address 127.0.0.1:39434)
0925 15:19:26.003007 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:26.003062 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:46039: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:19:26.003237 25901 catalog_manager.cc:4584] TS 240fa60c5f16431b9c0e39a77af00c3c (127.8.43.129:46039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1984bca0f9934ae593f20be0b73974d1: Network error: Client connection negotiation failed: client connection to 127.8.43.129:46039: connect: Connection reset by peer (error 104)
I20260925 15:19:26.003320 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 26106
I20260925 15:19:26.009029 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 26237
I20260925 15:19:26.014123 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 25884
2026-09-25T15:19:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (722 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:26 dist-test-slave-c5d6 krb5kdc[26407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:26 dist-test-slave-c5d6 krb5kdc[26407](info): set up 2 sockets
Sep 25 15:19:26 dist-test-slave-c5d6 krb5kdc[26407](info): commencing operation
krb5kdc: starting...
W20260925 15:19:27.047043 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:27.160221 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38485
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:38485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:27.239699 26420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:27.239917 26420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:27.239938 26420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:27.241014 26420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:27.241050 26420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:27.241063 26420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:27.241076 26420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:27.241088 26420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:27.242465 26420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:38485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:27.242653 26420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:27.242817 26420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:27.247534 26425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.247552 26426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.247552 26428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:27.248039 26420 server_base.cc:1034] running on GCE node
I20260925 15:19:27.248242 26420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:27.248490 26420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:27.249646 26420 hybrid_clock.cc:648] HybridClock initialized: now 1790349567249625 us; error 32 us; skew 500 ppm
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:27.252502 26420 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:27.252923 26420 webserver.cc:460] Webserver started at http://127.8.43.190:34925/ using document root <none> and password file <none>
I20260925 15:19:27.253106 26420 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:27.253149 26420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:27.253240 26420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:27.254292 26420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "9e99c33518094be5b9fe4255e47b1df4"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "dbf97af4416cade64af0ca0f3155d9b2"
server_key_iv: "eb6f5806637a96cd399ec376e91ed4df"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.254617 26420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "9e99c33518094be5b9fe4255e47b1df4"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "dbf97af4416cade64af0ca0f3155d9b2"
server_key_iv: "eb6f5806637a96cd399ec376e91ed4df"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.256039 26420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:27.256731 26435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.256940 26420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:27.257058 26420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "9e99c33518094be5b9fe4255e47b1df4"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "dbf97af4416cade64af0ca0f3155d9b2"
server_key_iv: "eb6f5806637a96cd399ec376e91ed4df"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.257149 26420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:27.277472 26420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:27.279780 26420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:27.279975 26420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:27.284587 26420 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:38485
I20260925 15:19:27.284670 26487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:38485 every 8 connection(s)
I20260925 15:19:27.285061 26420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:27.285996 26488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:27.288329 26488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4: Bootstrap starting.
I20260925 15:19:27.288887 26488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:27.289132 26488 log.cc:824] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:27.289897 26488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4: No bootstrap required, opened a new log
I20260925 15:19:27.291719 26488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e99c33518094be5b9fe4255e47b1df4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38485 } }
I20260925 15:19:27.291862 26488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:27.291893 26488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e99c33518094be5b9fe4255e47b1df4, State: Initialized, Role: FOLLOWER
I20260925 15:19:27.292047 26488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e99c33518094be5b9fe4255e47b1df4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38485 } }
I20260925 15:19:27.292145 26488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:27.292171 26488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:27.292214 26488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:27.292706 26488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e99c33518094be5b9fe4255e47b1df4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38485 } }
I20260925 15:19:27.292794 26488 leader_election.cc:302] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e99c33518094be5b9fe4255e47b1df4; no voters:
I20260925 15:19:27.292934 26488 leader_election.cc:288] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:27.293027 26491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:27.293187 26491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [term 1 LEADER]: Becoming Leader. State: Replica: 9e99c33518094be5b9fe4255e47b1df4, State: Running, Role: LEADER
I20260925 15:19:27.293201 26488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:27.293285 26491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e99c33518094be5b9fe4255e47b1df4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38485 } }
I20260925 15:19:27.293689 26493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e99c33518094be5b9fe4255e47b1df4. Latest consensus state: current_term: 1 leader_uuid: "9e99c33518094be5b9fe4255e47b1df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e99c33518094be5b9fe4255e47b1df4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38485 } } }
I20260925 15:19:27.293829 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e99c33518094be5b9fe4255e47b1df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e99c33518094be5b9fe4255e47b1df4" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38485 } } }
I20260925 15:19:27.293865 26493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:27.293941 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:27.294337 26500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:27.294670 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 26420
I20260925 15:19:27.294793 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:27.294905 26500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:27.295025 8366 external_mini_cluster.cc:1428] Setting key f1d350de6b4687cc60dae0251b7ff398
I20260925 15:19:27.296641 26500 catalog_manager.cc:1347] Generated new cluster ID: bc7fd8be3ec94298ab70390b1a82fc06
I20260925 15:19:27.296693 26500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:27.310601 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.295537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.8.43.190:38485)
0925 15:19:27.299560 (+ 4023us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.299569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.299600 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:19:27.299704 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.299710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.299771 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.299887 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:27.300809 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.302185 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.303310 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.303906 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.304011 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.306895 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:27.306900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:27.306900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:27.306914 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:27.308957 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.309337 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.309340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.309340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.309461 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.309664 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.309665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.309665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.309969 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:27.310115 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.310218 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.310352 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3965,"thread_start_us":83,"threads_started":1}
I20260925 15:19:27.319314 26500 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:27.320003 26500 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:27.327030 26500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4: Generated new TSK 0
I20260925 15:19:27.327214 26500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:19:27.356498 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38485
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:27.439525 26514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:27.439770 26514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:27.439810 26514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:27.440949 26514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:27.441017 26514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:27.441063 26514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:19:27.442557 26514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:27.442842 26514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:27.443073 26514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:27.448179 26522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.448172 26519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.448227 26520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:27.448532 26514 server_base.cc:1034] running on GCE node
I20260925 15:19:27.448772 26514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:27.448994 26514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:27.450155 26514 hybrid_clock.cc:648] HybridClock initialized: now 1790349567450148 us; error 38 us; skew 500 ppm
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:27.452708 26514 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:19:27.453125 26514 webserver.cc:460] Webserver started at http://127.8.43.129:45371/ using document root <none> and password file <none>
I20260925 15:19:27.453383 26514 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:27.453454 26514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:27.453601 26514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:27.454638 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "33e8020e34ca4ab8b835084e080465c0"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "5e86ff8ccdc1792af2dfda829e548b1f"
server_key_iv: "fbf9f5bc1cce672ba108822c8cb994fd"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.454998 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "33e8020e34ca4ab8b835084e080465c0"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "5e86ff8ccdc1792af2dfda829e548b1f"
server_key_iv: "fbf9f5bc1cce672ba108822c8cb994fd"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.456367 26514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:27.457093 26529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.457306 26514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:27.457401 26514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "33e8020e34ca4ab8b835084e080465c0"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "5e86ff8ccdc1792af2dfda829e548b1f"
server_key_iv: "fbf9f5bc1cce672ba108822c8cb994fd"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.457480 26514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:27.484584 26514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:27.486683 26514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:27.486865 26514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:27.487120 26514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:27.487479 26514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:27.487550 26514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.487586 26514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:27.487615 26514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.494144 26514 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:42781
I20260925 15:19:27.494184 26642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:42781 every 8 connection(s)
I20260925 15:19:27.494566 26514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:27.502106 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 26514
I20260925 15:19:27.502229 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:27.502480 8366 external_mini_cluster.cc:1428] Setting key 74acd5a6e7eb5300d8f5f0a8b47ea135
I20260925 15:19:27.504462 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.495379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:58495 (local address 127.8.43.190:38485)
0925 15:19:27.495465 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.495468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.495683 (+ 215us) server_negotiation.cc:407] Connection header received
0925 15:19:27.496203 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.496204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.496215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.496276 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:27.497184 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.497745 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.498780 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.499007 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.499067 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.501202 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:27.501204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:27.501204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:27.501211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:27.502791 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.503342 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.503343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.503344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.503402 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.503743 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.503744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.503744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.503874 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:27.503940 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.504329 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.504400 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:19:27.505018 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.495372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38485 (local address 127.8.43.129:58495)
0925 15:19:27.495563 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:27.495600 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:27.496114 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:27.496329 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:27.496332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:27.496887 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:27.497082 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.497087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.497867 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.497868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.498675 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.498678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.499086 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.499087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.499185 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.499650 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:27.499659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:27.501114 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:27.502870 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:27.502877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:27.502879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:27.503199 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:27.503485 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:27.503485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:27.503486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:27.503643 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:27.503982 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:27.503989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:27.504174 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 15:19:27.504308 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 15:19:27.504497 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260925 15:19:27.505601 26643 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38485
I20260925 15:19:27.505730 26643 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:27.505913 26643 heartbeater.cc:511] Master 127.8.43.190:38485 requested a full tablet report, sending...
I20260925 15:19:27.506517 26452 ts_manager.cc:194] Registered new tserver with Master: 33e8020e34ca4ab8b835084e080465c0 (127.8.43.129:42781)
I20260925 15:19:27.507429 26452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:58495
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:19:27.527390 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38485
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:27.609961 26648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:27.610190 26648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:27.610229 26648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:27.611650 26648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:27.611713 26648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:27.611765 26648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:19:27.614234 26648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:27.614486 26648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:27.614770 26648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:27.620254 26654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.620250 26656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.620251 26653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:27.620287 26648 server_base.cc:1034] running on GCE node
I20260925 15:19:27.620675 26648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:27.620924 26648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:27.622090 26648 hybrid_clock.cc:648] HybridClock initialized: now 1790349567622072 us; error 36 us; skew 500 ppm
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:27.624709 26648 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:19:27.625135 26648 webserver.cc:460] Webserver started at http://127.8.43.130:35727/ using document root <none> and password file <none>
I20260925 15:19:27.625365 26648 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:27.625432 26648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:27.625603 26648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:27.626613 26648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "4a8c19decca142039ae93bb8e00409b4"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "e173570788e0a4c742bfeaf92d40a0e2"
server_key_iv: "a08bcbc20cd6b17ce5131e3577257656"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.626974 26648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "4a8c19decca142039ae93bb8e00409b4"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "e173570788e0a4c742bfeaf92d40a0e2"
server_key_iv: "a08bcbc20cd6b17ce5131e3577257656"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.628340 26648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:27.629071 26663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.629314 26648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:27.629407 26648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "4a8c19decca142039ae93bb8e00409b4"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "e173570788e0a4c742bfeaf92d40a0e2"
server_key_iv: "a08bcbc20cd6b17ce5131e3577257656"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.629504 26648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:27.646611 26648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:27.648588 26648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:27.648746 26648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:27.648972 26648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:27.649267 26648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:27.649322 26648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.649362 26648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:27.649390 26648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.655249 26648 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:36079
I20260925 15:19:27.655292 26776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:36079 every 8 connection(s)
I20260925 15:19:27.655596 26648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:27.662148 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 26648
I20260925 15:19:27.662271 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:27.662463 8366 external_mini_cluster.cc:1428] Setting key cb597d2da2ca8eed6895c0d3076a8ac8
I20260925 15:19:27.665527 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.656325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:33737 (local address 127.8.43.190:38485)
0925 15:19:27.656385 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.656387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.656586 (+ 199us) server_negotiation.cc:407] Connection header received
0925 15:19:27.657105 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.657106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.657118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.657172 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:27.658076 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.658579 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.659668 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.659914 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.660000 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.662159 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:27.662160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:27.662161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:27.662167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:27.663862 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.664377 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.664378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.664378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.664449 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.664766 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.664768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.664768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.664903 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:27.665037 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.665329 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.665427 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:19:27.665722 26779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.656288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38485 (local address 127.8.43.130:33737)
0925 15:19:27.656468 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:27.656508 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:27.657004 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:27.657249 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:27.657252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:27.657778 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:27.657953 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.657958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.658682 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.658683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.659576 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.659579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.660007 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.660008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.660123 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.660623 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:27.660631 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:27.662029 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:27.663946 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:27.663953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:27.663955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:27.664277 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:27.664560 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:27.664561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:27.664561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:27.664682 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:27.665025 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:27.665032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:27.665173 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:19:27.665302 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:19:27.665484 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260925 15:19:27.666211 26777 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38485
I20260925 15:19:27.666303 26777 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:27.666512 26777 heartbeater.cc:511] Master 127.8.43.190:38485 requested a full tablet report, sending...
I20260925 15:19:27.666906 26452 ts_manager.cc:194] Registered new tserver with Master: 4a8c19decca142039ae93bb8e00409b4 (127.8.43.130:36079)
I20260925 15:19:27.667443 26452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:33737
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:19:27.687748 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38485
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:27.768548 26782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:27.768791 26782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:27.768832 26782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:27.770062 26782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:27.770112 26782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:27.770180 26782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:19:27.771679 26782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:27.771936 26782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:27.772173 26782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:27.777113 26788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:27.777189 26782 server_base.cc:1034] running on GCE node
W20260925 15:19:27.777119 26787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:27.777122 26790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:27.777509 26782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:27.777712 26782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:27.778901 26782 hybrid_clock.cc:648] HybridClock initialized: now 1790349567778880 us; error 36 us; skew 500 ppm
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:27.781904 26782 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:19:27.782462 26782 webserver.cc:460] Webserver started at http://127.8.43.131:36669/ using document root <none> and password file <none>
I20260925 15:19:27.782691 26782 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:27.782760 26782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:27.782914 26782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:27.784117 26782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "8c1cad4dae4a4692843b4291705b0cbc"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "2a15d3b846a20e216f3f35c00f306e2e"
server_key_iv: "af297e547b5d228fb2221dff82ff4eaa"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.784489 26782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "8c1cad4dae4a4692843b4291705b0cbc"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "2a15d3b846a20e216f3f35c00f306e2e"
server_key_iv: "af297e547b5d228fb2221dff82ff4eaa"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.785861 26782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:19:27.786619 26797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.786815 26782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.786928 26782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "8c1cad4dae4a4692843b4291705b0cbc"
format_stamp: "Formatted at 2026-09-25 15:19:27 on dist-test-slave-c5d6"
server_key: "2a15d3b846a20e216f3f35c00f306e2e"
server_key_iv: "af297e547b5d228fb2221dff82ff4eaa"
server_key_version: "encryptionkey@0"
I20260925 15:19:27.787034 26782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:27.806038 26782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:27.807927 26782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:27.808096 26782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:27.808317 26782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:27.808612 26782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:27.808665 26782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.808704 26782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:27.808733 26782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:27.814597 26782 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:35957
I20260925 15:19:27.814646 26910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:35957 every 8 connection(s)
I20260925 15:19:27.814931 26782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:19:27 dist-test-slave-c5d6 krb5kdc[26407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349567, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:27.822310 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 26782
I20260925 15:19:27.822417 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:27.822611 8366 external_mini_cluster.cc:1428] Setting key 003ff9926c88240b45151fea251a4404
I20260925 15:19:27.823674 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.815635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:46737 (local address 127.8.43.190:38485)
0925 15:19:27.815693 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.815695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.815862 (+ 167us) server_negotiation.cc:407] Connection header received
0925 15:19:27.816378 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.816379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.816388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.816463 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:27.817330 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.817819 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.818771 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.818945 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.818998 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.821062 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:27.821063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:27.821064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:27.821070 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:27.822217 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.822696 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.822697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.822698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.822759 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.823022 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.823023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.823023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.823138 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:27.823213 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.823509 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.823585 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:19:27.823849 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.815592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38485 (local address 127.8.43.131:46737)
0925 15:19:27.815737 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:27.815776 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:27.816298 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:27.816536 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:27.816539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:27.817074 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:27.817254 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.817259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.817902 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.817903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.818703 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.818705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.819010 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.819011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.819102 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.819567 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:27.819576 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:27.820973 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:27.822311 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:27.822317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:27.822319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:27.822609 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:27.822827 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:27.822827 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:27.822828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:27.822939 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:27.823205 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:27.823212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:27.823366 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 15:19:27.823478 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:19:27.823627 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260925 15:19:27.824323 26911 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38485
I20260925 15:19:27.824400 26911 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:27.824534 26911 heartbeater.cc:511] Master 127.8.43.190:38485 requested a full tablet report, sending...
I20260925 15:19:27.824855 26452 ts_manager.cc:194] Registered new tserver with Master: 8c1cad4dae4a4692843b4291705b0cbc (127.8.43.131:35957)
I20260925 15:19:27.825230 26452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:46737
I20260925 15:19:27.834290 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:27.841234 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.835638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.8.43.190:38485)
0925 15:19:27.835708 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.835710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.835816 (+ 106us) server_negotiation.cc:407] Connection header received
0925 15:19:27.835853 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.835853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.835861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.835905 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:27.836377 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.836825 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.837517 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.837680 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.837726 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.838982 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:27.838983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:27.838983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:27.838988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:27.840191 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.840544 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.840545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.840545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.840592 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:27.840765 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:27.840766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:27.840766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:27.840877 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:27.840949 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.841060 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.841103 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:19:27.842845 26452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:27.844450 26452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:27.850818 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.847091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.8.43.129:42781)
0925 15:19:27.847277 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.847281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.847292 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:27.847391 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.847393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.847409 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.847651 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:27.847727 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.848783 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.849933 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.850404 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.850493 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.850549 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.850703 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.850744 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260925 15:19:27.850811 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.846665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:42781 (local address 127.0.0.1:43354)
0925 15:19:27.846923 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:27.846933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:27.847009 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:27.847507 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:27.847508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:27.847511 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:27.847577 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.847579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.848892 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.848893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.849844 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.849846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.850501 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.850502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.850577 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.850596 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 15:19:27.850683 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:19:27.850730 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260925 15:19:27.850816 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.846664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:36079 (local address 127.0.0.1:38776)
0925 15:19:27.846822 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:27.846859 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:27.846926 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:27.847454 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:27.847455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:27.847457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:27.847547 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.847549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.848420 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.848421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.849567 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.849570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.850513 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.850514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.850599 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.850603 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:27.850660 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:19:27.850726 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":4,"thread_start_us":55,"threads_started":1}
I20260925 15:19:27.850965 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.846675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:35957 (local address 127.0.0.1:35752)
0925 15:19:27.847085 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:27.847091 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:27.847126 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:27.847326 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:27.847329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:27.847332 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:27.847618 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.847620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.848926 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.848927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.849893 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:27.849896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.850727 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.850728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:27.850795 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.850798 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:27.850825 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 15:19:27.850859 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":74,"threads_started":1}
I20260925 15:19:27.851305 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.846889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.8.43.131:35957)
0925 15:19:27.847055 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.847059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.847129 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:19:27.847191 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.847193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.847209 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.847315 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:27.847696 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.848799 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.849993 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.850643 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.851077 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.851143 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.851168 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.851207 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260925 15:19:27.851462 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:27.846761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.8.43.130:36079)
0925 15:19:27.847311 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:27.847315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:27.847326 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:27.847364 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:27.847366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:27.847382 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:27.847454 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:27.847630 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.848308 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.849658 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:27.850349 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:27.850414 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:27.850468 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:27.850856 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 15:19:27.851385 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":81,"threads_started":1}
I20260925 15:19:27.851643 26577 tablet_service.cc:1467] Processing CreateTablet for tablet 405667220ea74291a552670f5a984900 (DEFAULT_TABLE table=test-table [id=d090d9508191410791f446199a765134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:27.851866 26845 tablet_service.cc:1467] Processing CreateTablet for tablet 405667220ea74291a552670f5a984900 (DEFAULT_TABLE table=test-table [id=d090d9508191410791f446199a765134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:27.851979 26577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405667220ea74291a552670f5a984900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:27.852137 26845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405667220ea74291a552670f5a984900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:27.852077 26711 tablet_service.cc:1467] Processing CreateTablet for tablet 405667220ea74291a552670f5a984900 (DEFAULT_TABLE table=test-table [id=d090d9508191410791f446199a765134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:27.852309 26711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405667220ea74291a552670f5a984900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:27.854835 26928 tablet_bootstrap.cc:492] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Bootstrap starting.
I20260925 15:19:27.855350 26929 tablet_bootstrap.cc:492] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0: Bootstrap starting.
I20260925 15:19:27.855541 26928 tablet_bootstrap.cc:654] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:27.856029 26929 tablet_bootstrap.cc:654] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:27.856123 26928 log.cc:824] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:27.856606 26929 log.cc:824] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:27.855199 26930 tablet_bootstrap.cc:492] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Bootstrap starting.
I20260925 15:19:27.856906 26928 tablet_bootstrap.cc:492] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: No bootstrap required, opened a new log
I20260925 15:19:27.856978 26928 ts_tablet_manager.cc:1397] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:27.857425 26930 tablet_bootstrap.cc:654] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:27.857928 26930 log.cc:824] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:27.858425 26929 tablet_bootstrap.cc:492] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0: No bootstrap required, opened a new log
I20260925 15:19:27.858500 26929 ts_tablet_manager.cc:1397] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:19:27.858556 26928 raft_consensus.cc:348] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:27.858709 26928 raft_consensus.cc:374] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:27.858749 26928 raft_consensus.cc:729] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8c19decca142039ae93bb8e00409b4, State: Initialized, Role: FOLLOWER
I20260925 15:19:27.858883 26928 consensus_queue.cc:260] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:27.859130 26928 ts_tablet_manager.cc:1428] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:27.859164 26777 heartbeater.cc:503] Master 127.8.43.190:38485 was elected leader, sending a full tablet report...
I20260925 15:19:27.860538 26929 raft_consensus.cc:348] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:27.860679 26929 raft_consensus.cc:374] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:27.860723 26929 raft_consensus.cc:729] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e8020e34ca4ab8b835084e080465c0, State: Initialized, Role: FOLLOWER
I20260925 15:19:27.860824 26929 consensus_queue.cc:260] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:27.861109 26929 ts_tablet_manager.cc:1428] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:27.861116 26643 heartbeater.cc:503] Master 127.8.43.190:38485 was elected leader, sending a full tablet report...
I20260925 15:19:27.861723 26930 tablet_bootstrap.cc:492] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: No bootstrap required, opened a new log
I20260925 15:19:27.861783 26930 ts_tablet_manager.cc:1397] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 15:19:27.863404 26930 raft_consensus.cc:348] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:27.863524 26930 raft_consensus.cc:374] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:27.863545 26930 raft_consensus.cc:729] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1cad4dae4a4692843b4291705b0cbc, State: Initialized, Role: FOLLOWER
I20260925 15:19:27.863670 26930 consensus_queue.cc:260] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:27.863914 26930 ts_tablet_manager.cc:1428] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:27.863972 26911 heartbeater.cc:503] Master 127.8.43.190:38485 was elected leader, sending a full tablet report...
W20260925 15:19:27.906201 26778 tablet.cc:2369] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:19:27.995373 26644 tablet.cc:2369] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:28.049357 26936 raft_consensus.cc:482] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:28.049487 26936 raft_consensus.cc:504] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:28.049911 26936 leader_election.cc:288] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33e8020e34ca4ab8b835084e080465c0 (127.8.43.129:42781), 4a8c19decca142039ae93bb8e00409b4 (127.8.43.130:36079)
I20260925 15:19:28.052809 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.050215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:51179 (local address 127.8.43.130:36079)
0925 15:19:28.050258 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.050260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.050271 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:28.050343 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.050344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.050356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.050443 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:28.050622 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.051095 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.052079 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.052449 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.052537 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.052546 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.052703 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.052738 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:19:28.052809 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.050122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:36079 (local address 127.8.43.131:51179)
0925 15:19:28.050189 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.050202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.050270 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.050452 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.050453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.050455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:28.050542 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.050544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.051175 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.051176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.051987 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.051989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.052534 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.052535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.052617 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.052620 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.052695 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.052737 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:19:28.052898 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.050246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:55787 (local address 127.8.43.129:42781)
0925 15:19:28.050301 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.050303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.050447 (+ 144us) server_negotiation.cc:407] Connection header received
0925 15:19:28.050531 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.050532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.050545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.050613 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:28.050760 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.051259 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.052205 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.052560 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.052639 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.052646 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.052799 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.052834 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:19:28.052910 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.050133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:42781 (local address 127.8.43.131:55787)
0925 15:19:28.050375 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.050383 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.050462 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.050616 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.050616 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.050618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:28.050694 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.050695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.051356 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.051356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.052109 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.052111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.052638 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.052638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.052714 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.052718 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.052791 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.052829 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":90,"thread_start_us":91,"threads_started":1}
I20260925 15:19:28.053174 26731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405667220ea74291a552670f5a984900" candidate_uuid: "8c1cad4dae4a4692843b4291705b0cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a8c19decca142039ae93bb8e00409b4" is_pre_election: true
I20260925 15:19:28.053349 26731 raft_consensus.cc:2454] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1cad4dae4a4692843b4291705b0cbc in term 0.
I20260925 15:19:28.053387 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405667220ea74291a552670f5a984900" candidate_uuid: "8c1cad4dae4a4692843b4291705b0cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33e8020e34ca4ab8b835084e080465c0" is_pre_election: true
I20260925 15:19:28.053550 26597 raft_consensus.cc:2454] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1cad4dae4a4692843b4291705b0cbc in term 0.
I20260925 15:19:28.053601 26799 leader_election.cc:302] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a8c19decca142039ae93bb8e00409b4, 8c1cad4dae4a4692843b4291705b0cbc; no voters:
I20260925 15:19:28.053733 26936 raft_consensus.cc:2781] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:28.053782 26936 raft_consensus.cc:482] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:28.053804 26936 raft_consensus.cc:3037] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:28.054541 26936 raft_consensus.cc:504] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:28.054678 26936 leader_election.cc:288] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [CANDIDATE]: Term 1 election: Requested vote from peers 33e8020e34ca4ab8b835084e080465c0 (127.8.43.129:42781), 4a8c19decca142039ae93bb8e00409b4 (127.8.43.130:36079)
I20260925 15:19:28.054829 26731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405667220ea74291a552670f5a984900" candidate_uuid: "8c1cad4dae4a4692843b4291705b0cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a8c19decca142039ae93bb8e00409b4"
I20260925 15:19:28.054843 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405667220ea74291a552670f5a984900" candidate_uuid: "8c1cad4dae4a4692843b4291705b0cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33e8020e34ca4ab8b835084e080465c0"
I20260925 15:19:28.054909 26597 raft_consensus.cc:3037] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:28.054909 26731 raft_consensus.cc:3037] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:28.055806 26731 raft_consensus.cc:2454] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1cad4dae4a4692843b4291705b0cbc in term 1.
I20260925 15:19:28.055806 26597 raft_consensus.cc:2454] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1cad4dae4a4692843b4291705b0cbc in term 1.
I20260925 15:19:28.055975 26799 leader_election.cc:302] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a8c19decca142039ae93bb8e00409b4, 8c1cad4dae4a4692843b4291705b0cbc; no voters:
I20260925 15:19:28.056067 26936 raft_consensus.cc:2781] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:28.056214 26936 raft_consensus.cc:686] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 1 LEADER]: Becoming Leader. State: Replica: 8c1cad4dae4a4692843b4291705b0cbc, State: Running, Role: LEADER
I20260925 15:19:28.056437 26936 consensus_queue.cc:237] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } }
I20260925 15:19:28.057327 26452 catalog_manager.cc:5495] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c1cad4dae4a4692843b4291705b0cbc (127.8.43.131). New cstate: current_term: 1 leader_uuid: "8c1cad4dae4a4692843b4291705b0cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c1cad4dae4a4692843b4291705b0cbc" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 35957 } health_report { overall_health: HEALTHY } } }
W20260925 15:19:28.065471 26912 tablet.cc:2369] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:28.075872 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.073666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.8.43.131:35957)
0925 15:19:28.073734 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.073736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.073743 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:19:28.073772 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.073772 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.073787 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.073845 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:28.073962 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.074419 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.075115 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.075357 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.075408 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.075552 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:28.075634 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:28.075706 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.075749 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.075755 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:19:28.077646 26731 raft_consensus.cc:1261] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 1 FOLLOWER]: Refusing update from remote peer 8c1cad4dae4a4692843b4291705b0cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:28.077647 26597 raft_consensus.cc:1261] T 405667220ea74291a552670f5a984900 P 33e8020e34ca4ab8b835084e080465c0 [term 1 FOLLOWER]: Refusing update from remote peer 8c1cad4dae4a4692843b4291705b0cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:28.077968 26938 consensus_queue.cc:1035] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "33e8020e34ca4ab8b835084e080465c0" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 42781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:28.078092 26936 consensus_queue.cc:1035] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a8c19decca142039ae93bb8e00409b4" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:28.080437 26941 mvcc.cc:205] Tried to move back new op lower bound from 7333271830843088896 to 7333271830759960576. Current Snapshot: MvccSnapshot[applied={T|T < 7333271830843088896}]
I20260925 15:19:28.080670 26943 mvcc.cc:205] Tried to move back new op lower bound from 7333271830843088896 to 7333271830759960576. Current Snapshot: MvccSnapshot[applied={T|T < 7333271830843088896}]
I20260925 15:19:28.081034 26942 mvcc.cc:205] Tried to move back new op lower bound from 7333271830843088896 to 7333271830759960576. Current Snapshot: MvccSnapshot[applied={T|T < 7333271830843088896}]
I20260925 15:19:28.083155 26452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:28.083302 26452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:28.084079 26452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e99c33518094be5b9fe4255e47b1df4: Sending DeleteTablet for 3 replicas of tablet 405667220ea74291a552670f5a984900
I20260925 15:19:28.084414 26845 tablet_service.cc:1514] Processing DeleteTablet for tablet 405667220ea74291a552670f5a984900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:28 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:35752
I20260925 15:19:28.084436 26711 tablet_service.cc:1514] Processing DeleteTablet for tablet 405667220ea74291a552670f5a984900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:28 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:38776
I20260925 15:19:28.084656 26951 tablet_replica.cc:330] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: stopping tablet replica
I20260925 15:19:28.084638 26577 tablet_service.cc:1514] Processing DeleteTablet for tablet 405667220ea74291a552670f5a984900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:28 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:43354
I20260925 15:19:28.084789 26951 raft_consensus.cc:2229] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:28.084940 26951 raft_consensus.cc:2258] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:28.084970 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 26514
I20260925 15:19:28.086661 26951 ts_tablet_manager.cc:1905] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:28.088560 26951 ts_tablet_manager.cc:1918] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:28.088649 26951 log.cc:1196] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/405667220ea74291a552670f5a984900
I20260925 15:19:28.088891 26951 ts_tablet_manager.cc:1939] T 405667220ea74291a552670f5a984900 P 4a8c19decca142039ae93bb8e00409b4: Deleting consensus metadata
I20260925 15:19:28.089411 26437 catalog_manager.cc:4841] TS 4a8c19decca142039ae93bb8e00409b4 (127.8.43.130:36079): tablet 405667220ea74291a552670f5a984900 (table test-table [id=d090d9508191410791f446199a765134]) successfully deleted
W20260925 15:19:28.090960 26438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:42781 (error 108)
I20260925 15:19:28.091173 26950 tablet_replica.cc:330] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: stopping tablet replica
I20260925 15:19:28.091297 26950 raft_consensus.cc:2229] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:28.091425 26950 raft_consensus.cc:2258] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:28.091511 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.091396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:42781 (local address 127.0.0.1:43356)
0925 15:19:28.091444 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.091478 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:42781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:19:28.091686 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 26648
I20260925 15:19:28.091846 26950 ts_tablet_manager.cc:1905] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:28.093348 26950 ts_tablet_manager.cc:1918] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:28.093431 26950 log.cc:1196] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/405667220ea74291a552670f5a984900
I20260925 15:19:28.093694 26950 ts_tablet_manager.cc:1939] T 405667220ea74291a552670f5a984900 P 8c1cad4dae4a4692843b4291705b0cbc: Deleting consensus metadata
I20260925 15:19:28.094157 26439 catalog_manager.cc:4841] TS 8c1cad4dae4a4692843b4291705b0cbc (127.8.43.131:35957): tablet 405667220ea74291a552670f5a984900 (table test-table [id=d090d9508191410791f446199a765134]) successfully deleted
W20260925 15:19:28.095690 26438 catalog_manager.cc:4584] TS 33e8020e34ca4ab8b835084e080465c0 (127.8.43.129:42781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 405667220ea74291a552670f5a984900: Network error: Client connection negotiation failed: client connection to 127.8.43.129:42781: connect: Connection refused (error 111)
I20260925 15:19:28.099090 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 26782
I20260925 15:19:28.103751 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 26420
2026-09-25T15:19:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2090 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:19:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:28Z Disabled control of system clock
I20260925 15:19:28.131006 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:41137
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:41137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:19:28.210512 26957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:28.210726 26957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:28.210747 26957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:28.211894 26957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:28.211934 26957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:28.211947 26957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:28.211961 26957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:28.211972 26957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:28.213222 26957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:41137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:28.213407 26957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:28.213639 26957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:28.218385 26962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.218376 26965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:28.218406 26957 server_base.cc:1034] running on GCE node
W20260925 15:19:28.218376 26963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:28.218783 26957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:28.219107 26957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:28.220270 26957 hybrid_clock.cc:648] HybridClock initialized: now 1790349568220244 us; error 54 us; skew 500 ppm
I20260925 15:19:28.221460 26957 webserver.cc:460] Webserver started at http://127.8.43.190:38867/ using document root <none> and password file <none>
I20260925 15:19:28.221709 26957 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:28.221776 26957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:28.221910 26957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:28.222951 26957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "e05bbb43828546518de71e6a09f71dab"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "cfe1d773895137ff3e79b5cb3fd9b543"
server_key_iv: "c826067df42c8e616b2ab5069c9ee9b8"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.223363 26957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "e05bbb43828546518de71e6a09f71dab"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "cfe1d773895137ff3e79b5cb3fd9b543"
server_key_iv: "c826067df42c8e616b2ab5069c9ee9b8"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.224681 26957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:28.225474 26971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.225641 26957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:28.225737 26957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "e05bbb43828546518de71e6a09f71dab"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "cfe1d773895137ff3e79b5cb3fd9b543"
server_key_iv: "c826067df42c8e616b2ab5069c9ee9b8"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.225824 26957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:28.242245 26957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:28.242554 26957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:28.242705 26957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:28.246613 26957 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:41137
I20260925 15:19:28.246659 27023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:41137 every 8 connection(s)
I20260925 15:19:28.247021 26957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:28.248111 27024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:28.250470 27024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab: Bootstrap starting.
I20260925 15:19:28.251082 27024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:28.251432 27024 log.cc:824] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:28.252166 27024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab: No bootstrap required, opened a new log
I20260925 15:19:28.253793 27024 raft_consensus.cc:348] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05bbb43828546518de71e6a09f71dab" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41137 } }
I20260925 15:19:28.253943 27024 raft_consensus.cc:374] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:28.253970 27024 raft_consensus.cc:729] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05bbb43828546518de71e6a09f71dab, State: Initialized, Role: FOLLOWER
I20260925 15:19:28.254146 27024 consensus_queue.cc:260] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05bbb43828546518de71e6a09f71dab" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41137 } }
I20260925 15:19:28.254251 27024 raft_consensus.cc:388] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:28.254297 27024 raft_consensus.cc:482] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:28.254354 27024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:28.254930 27024 raft_consensus.cc:504] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05bbb43828546518de71e6a09f71dab" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41137 } }
I20260925 15:19:28.255043 27024 leader_election.cc:302] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05bbb43828546518de71e6a09f71dab; no voters:
I20260925 15:19:28.255283 27024 leader_election.cc:288] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:28.255304 27027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:28.255425 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 26957
I20260925 15:19:28.255450 27027 raft_consensus.cc:686] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [term 1 LEADER]: Becoming Leader. State: Replica: e05bbb43828546518de71e6a09f71dab, State: Running, Role: LEADER
I20260925 15:19:28.255543 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:28.255544 27027 consensus_queue.cc:237] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05bbb43828546518de71e6a09f71dab" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41137 } }
I20260925 15:19:28.255564 27024 sys_catalog.cc:564] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:28.255743 8366 external_mini_cluster.cc:1428] Setting key e5cbfd59a37b1dd514539fe115f39f69
I20260925 15:19:28.255926 27028 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e05bbb43828546518de71e6a09f71dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05bbb43828546518de71e6a09f71dab" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41137 } } }
I20260925 15:19:28.256021 27028 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:28.256206 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader e05bbb43828546518de71e6a09f71dab. Latest consensus state: current_term: 1 leader_uuid: "e05bbb43828546518de71e6a09f71dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05bbb43828546518de71e6a09f71dab" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41137 } } }
I20260925 15:19:28.256284 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:28.256374 27031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:28.256981 27031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:28.259112 27031 catalog_manager.cc:1347] Generated new cluster ID: 133d730b12524efea64bae33c287302e
I20260925 15:19:28.259167 27031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:28.262614 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.256378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.8.43.190:41137)
0925 15:19:28.256554 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.256563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.257188 (+ 625us) server_negotiation.cc:407] Connection header received
0925 15:19:28.257624 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.257636 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.257745 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.257926 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:28.258148 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.259339 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.260434 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.261035 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.261146 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.261885 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:28.261889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:28.261890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:28.262101 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:28.262145 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:28.262158 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:28.262227 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.262315 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.262390 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260925 15:19:28.278048 27031 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:28.278767 27031 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:28.288658 27031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab: Generated new TSK 0
I20260925 15:19:28.288872 27031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:28.294049 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:41137
--builtin_ntp_servers=127.8.43.148:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:19:28.381567 27048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:28.381811 27048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:28.381850 27048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:28.383015 27048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:28.383083 27048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:28.383128 27048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:19:28.384507 27048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:41137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:28.384773 27048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:28.385128 27048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:28.390261 27054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.390308 27056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.390269 27053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:28.390434 27048 server_base.cc:1034] running on GCE node
I20260925 15:19:28.390777 27048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:28.391047 27048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:28.392226 27048 hybrid_clock.cc:648] HybridClock initialized: now 1790349568392196 us; error 58 us; skew 500 ppm
I20260925 15:19:28.393676 27048 webserver.cc:460] Webserver started at http://127.8.43.129:45097/ using document root <none> and password file <none>
I20260925 15:19:28.393988 27048 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:28.394069 27048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:28.394248 27048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:28.395505 27048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "db1e66f02adf4624928da3a604a972bd"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "52d8a842b9d525f4c33a53aef9bd477c"
server_key_iv: "8a51ebd71280698ac0adec12bd856862"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.395893 27048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "db1e66f02adf4624928da3a604a972bd"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "52d8a842b9d525f4c33a53aef9bd477c"
server_key_iv: "8a51ebd71280698ac0adec12bd856862"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.397369 27048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:28.398224 27062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.398470 27048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:28.398597 27048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "db1e66f02adf4624928da3a604a972bd"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "52d8a842b9d525f4c33a53aef9bd477c"
server_key_iv: "8a51ebd71280698ac0adec12bd856862"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.398685 27048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:28.423264 27048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:28.423542 27048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:28.423647 27048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:28.423842 27048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:28.424175 27048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:28.424211 27048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.424233 27048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:28.424247 27048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.430306 27048 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:46389
I20260925 15:19:28.430388 27175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:46389 every 8 connection(s)
I20260925 15:19:28.430696 27048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:28.435958 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.431469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:58969 (local address 127.8.43.190:41137)
0925 15:19:28.431541 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.431542 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.431779 (+ 237us) server_negotiation.cc:407] Connection header received
0925 15:19:28.432319 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.432321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.432335 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.432392 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:28.433051 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.433555 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.434549 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.434742 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.434826 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.435667 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:28.435668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:28.435668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:28.435673 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:28.435687 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:28.435691 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:28.435751 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.435849 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.435882 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:19:28.436190 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.431469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:41137 (local address 127.8.43.129:58969)
0925 15:19:28.431656 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.431690 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.432213 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.432464 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.432467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.432474 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:28.432956 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.432960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.433647 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.433648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.434425 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.434427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.434837 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.434838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.434966 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.435525 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:28.435525 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:28.435553 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:28.435574 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:28.435765 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:28.435773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:28.435779 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.435854 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.435950 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260925 15:19:28.436677 27176 heartbeater.cc:347] Connected to a master server at 127.8.43.190:41137
I20260925 15:19:28.436781 27176 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:28.436950 27176 heartbeater.cc:511] Master 127.8.43.190:41137 requested a full tablet report, sending...
I20260925 15:19:28.437436 26988 ts_manager.cc:194] Registered new tserver with Master: db1e66f02adf4624928da3a604a972bd (127.8.43.129:46389)
I20260925 15:19:28.437975 26988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.129:58969
I20260925 15:19:28.439441 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 27048
I20260925 15:19:28.439527 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:28.439693 8366 external_mini_cluster.cc:1428] Setting key 78f2826893ff0fdee9107984d3976d56
I20260925 15:19:28.440526 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:41137
--builtin_ntp_servers=127.8.43.148:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:19:28.519742 27179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:28.519930 27179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:28.519950 27179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:28.520989 27179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:28.521020 27179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:28.521044 27179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:19:28.522266 27179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:41137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:28.522436 27179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:28.522667 27179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:28.527486 27187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.527555 27184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.527509 27185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:28.527673 27179 server_base.cc:1034] running on GCE node
I20260925 15:19:28.527890 27179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:28.528141 27179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:28.529333 27179 hybrid_clock.cc:648] HybridClock initialized: now 1790349568529315 us; error 27 us; skew 500 ppm
I20260925 15:19:28.533946 27179 webserver.cc:460] Webserver started at http://127.8.43.130:33345/ using document root <none> and password file <none>
I20260925 15:19:28.534226 27179 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:28.534303 27179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:28.534489 27179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:28.535734 27179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "754dd5bfbf764847acea0755364176f1"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "3963bc3becd42079aa33b3e146452b2a"
server_key_iv: "e77f809ce8059c776d285b89ea44d3ec"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.536113 27179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "754dd5bfbf764847acea0755364176f1"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "3963bc3becd42079aa33b3e146452b2a"
server_key_iv: "e77f809ce8059c776d285b89ea44d3ec"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.537473 27179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:28.538276 27193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.538470 27179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:28.538548 27179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "754dd5bfbf764847acea0755364176f1"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "3963bc3becd42079aa33b3e146452b2a"
server_key_iv: "e77f809ce8059c776d285b89ea44d3ec"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.538594 27179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:28.545665 27179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:28.545874 27179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:28.545966 27179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:28.546114 27179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:28.546343 27179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:28.546370 27179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.546393 27179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:28.546411 27179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.551913 27179 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:32929
I20260925 15:19:28.551980 27306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:32929 every 8 connection(s)
I20260925 15:19:28.552278 27179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:28.554859 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 27179
I20260925 15:19:28.554970 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:28.555194 8366 external_mini_cluster.cc:1428] Setting key 13499611c6fe0a53801999cb6c6f0100
I20260925 15:19:28.556278 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:41137
--builtin_ntp_servers=127.8.43.148:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:19:28.558090 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.553020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:38865 (local address 127.8.43.190:41137)
0925 15:19:28.553117 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.553120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.553305 (+ 185us) server_negotiation.cc:407] Connection header received
0925 15:19:28.553846 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.553848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.553861 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.553919 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:28.554609 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.555205 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.556468 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.556786 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.556860 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.557759 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:28.557760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:28.557761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:28.557766 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:28.557780 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:28.557785 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:28.557839 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.557976 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.558007 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:19:28.558348 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.552926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:41137 (local address 127.8.43.130:38865)
0925 15:19:28.553177 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.553222 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.553734 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.553985 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.553989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.553999 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:28.554500 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.554505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.555348 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.555349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.556376 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.556380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.556872 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.556873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.557013 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.557560 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:28.557560 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:28.557589 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:28.557635 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:28.557840 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:28.557845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:28.557848 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.557981 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.558087 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":117,"threads_started":1}
I20260925 15:19:28.558813 27307 heartbeater.cc:347] Connected to a master server at 127.8.43.190:41137
I20260925 15:19:28.558904 27307 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:28.559118 27307 heartbeater.cc:511] Master 127.8.43.190:41137 requested a full tablet report, sending...
I20260925 15:19:28.559548 26988 ts_manager.cc:194] Registered new tserver with Master: 754dd5bfbf764847acea0755364176f1 (127.8.43.130:32929)
I20260925 15:19:28.560005 26988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.130:38865
W20260925 15:19:28.636649 27310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:28.636895 27310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:28.636931 27310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:28.638074 27310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:28.638135 27310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:28.638176 27310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:19:28.639552 27310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:41137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:28.639804 27310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:28.640105 27310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:28.645084 27315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.645084 27318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:28.645087 27316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:28.645440 27310 server_base.cc:1034] running on GCE node
I20260925 15:19:28.645721 27310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:28.645958 27310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:28.647153 27310 hybrid_clock.cc:648] HybridClock initialized: now 1790349568647137 us; error 32 us; skew 500 ppm
I20260925 15:19:28.648519 27310 webserver.cc:460] Webserver started at http://127.8.43.131:39645/ using document root <none> and password file <none>
I20260925 15:19:28.648769 27310 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:28.648842 27310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:28.648975 27310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:28.650063 27310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "b75765063ff04d70a759e5725b2c3cb8"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "a24b1630e3e23682c008bc366762e5d7"
server_key_iv: "d1b051151c56963621572b8507bbf64e"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.650491 27310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "b75765063ff04d70a759e5725b2c3cb8"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "a24b1630e3e23682c008bc366762e5d7"
server_key_iv: "d1b051151c56963621572b8507bbf64e"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.651882 27310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:28.652629 27324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.652801 27310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:28.652904 27310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "b75765063ff04d70a759e5725b2c3cb8"
format_stamp: "Formatted at 2026-09-25 15:19:28 on dist-test-slave-c5d6"
server_key: "a24b1630e3e23682c008bc366762e5d7"
server_key_iv: "d1b051151c56963621572b8507bbf64e"
server_key_version: "encryptionkey@0"
I20260925 15:19:28.652995 27310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:28.661381 27310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:28.661659 27310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:28.661799 27310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:28.662017 27310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:28.662325 27310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:28.662377 27310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.662416 27310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:28.662443 27310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:28.668325 27310 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:36885
I20260925 15:19:28.668371 27437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:36885 every 8 connection(s)
I20260925 15:19:28.668707 27310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:28.671380 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 27310
I20260925 15:19:28.671455 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:28.671681 8366 external_mini_cluster.cc:1428] Setting key 88613c1ac9c81ca8ea22961c4d48cffd
I20260925 15:19:28.674113 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.669437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:52879 (local address 127.8.43.190:41137)
0925 15:19:28.669513 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.669514 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.669755 (+ 241us) server_negotiation.cc:407] Connection header received
0925 15:19:28.670230 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.670231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.670240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.670300 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:28.670926 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.671575 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.672915 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.673107 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.673159 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.673903 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:28.673904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:28.673905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:28.673909 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:28.673921 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:28.673924 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:28.673965 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.674032 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.674056 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:19:28.674311 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.669417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:41137 (local address 127.8.43.131:52879)
0925 15:19:28.669622 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.669664 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.670150 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.670353 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.670357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.670365 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:28.670859 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.670863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.671663 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.671664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.672846 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.672848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.673175 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.673176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.673284 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.673782 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:28.673783 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:28.673809 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:28.673830 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:28.673976 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:28.673979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:28.673982 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.674035 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.674111 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260925 15:19:28.674691 27438 heartbeater.cc:347] Connected to a master server at 127.8.43.190:41137
I20260925 15:19:28.674779 27438 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:28.674949 27438 heartbeater.cc:511] Master 127.8.43.190:41137 requested a full tablet report, sending...
I20260925 15:19:28.675303 26988 ts_manager.cc:194] Registered new tserver with Master: b75765063ff04d70a759e5725b2c3cb8 (127.8.43.131:36885)
I20260925 15:19:28.675632 26988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.131:52879
I20260925 15:19:28.683663 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:28.687597 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.684985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.8.43.190:41137)
0925 15:19:28.685043 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.685045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.685164 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:19:28.685212 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.685213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.685223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.685273 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:28.685392 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.685831 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.686526 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.686701 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.686755 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.687384 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:28.687385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:28.687386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:28.687389 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:28.687401 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:28.687405 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:28.687471 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.687538 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.687547 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:19:28.688943 26988 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:28.690392 26988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:28.696389 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.692454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46389 (local address 127.0.0.1:47326)
0925 15:19:28.692740 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.692749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.692786 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.692975 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.693003 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.693006 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:28.693069 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.693071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.694162 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.694163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.695197 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.695200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.696113 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.696114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.696187 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.696199 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.696272 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.696318 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":81,"threads_started":1}
I20260925 15:19:28.696403 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.692571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.8.43.129:46389)
0925 15:19:28.692787 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.692792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.692805 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:19:28.692848 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.692850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.692865 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.692954 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:28.693153 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.694083 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.695316 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.696020 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.696110 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.696203 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.696290 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.696326 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260925 15:19:28.696841 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.692607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:36885 (local address 127.0.0.1:32774)
0925 15:19:28.692944 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.692954 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.693018 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.693316 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.693317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.693319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:28.693376 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.693378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.694701 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.694702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.695628 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.695630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.696616 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.696617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.696680 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.696683 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.696736 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.696772 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260925 15:19:28.696869 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.692768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.8.43.131:36885)
0925 15:19:28.692905 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.692909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.693011 (+ 102us) server_negotiation.cc:407] Connection header received
0925 15:19:28.693082 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.693084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.693100 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.693471 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:28.693542 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.694601 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.695710 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.696413 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.696487 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.696583 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.696754 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.696795 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260925 15:19:28.697122 27110 tablet_service.cc:1467] Processing CreateTablet for tablet 32b562176a6d4fdfb39137c4bf0ac231 (DEFAULT_TABLE table=test-table [id=4a87f9885b49418994aa74f30c932235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:28.697443 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.692400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:32929 (local address 127.0.0.1:34784)
0925 15:19:28.692597 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.692639 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.692699 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.693299 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.693300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.693302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:28.693374 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.693376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.694690 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.694691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.695795 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.695797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.697215 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.697216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.697280 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.697284 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.697342 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.697381 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":55,"thread_start_us":78,"threads_started":1}
I20260925 15:19:28.697525 27110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b562176a6d4fdfb39137c4bf0ac231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:28.697535 27372 tablet_service.cc:1467] Processing CreateTablet for tablet 32b562176a6d4fdfb39137c4bf0ac231 (DEFAULT_TABLE table=test-table [id=4a87f9885b49418994aa74f30c932235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:28.697674 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.692587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.8.43.130:32929)
0925 15:19:28.693084 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.693088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.693096 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:19:28.693150 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.693152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.693166 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.693240 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:28.693459 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.694604 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.695877 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.696878 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.697198 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.697284 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.697553 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.697595 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":79,"threads_started":1}
I20260925 15:19:28.697865 27372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b562176a6d4fdfb39137c4bf0ac231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:28.699894 27241 tablet_service.cc:1467] Processing CreateTablet for tablet 32b562176a6d4fdfb39137c4bf0ac231 (DEFAULT_TABLE table=test-table [id=4a87f9885b49418994aa74f30c932235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:28.700215 27241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b562176a6d4fdfb39137c4bf0ac231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:28.700387 27452 tablet_bootstrap.cc:492] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd: Bootstrap starting.
I20260925 15:19:28.701054 27452 tablet_bootstrap.cc:654] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:28.701354 27452 log.cc:824] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:28.702099 27452 tablet_bootstrap.cc:492] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd: No bootstrap required, opened a new log
I20260925 15:19:28.702150 27452 ts_tablet_manager.cc:1397] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:28.702998 27455 tablet_bootstrap.cc:492] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: Bootstrap starting.
I20260925 15:19:28.703086 27453 tablet_bootstrap.cc:492] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: Bootstrap starting.
I20260925 15:19:28.703745 27455 tablet_bootstrap.cc:654] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:28.703745 27453 tablet_bootstrap.cc:654] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:28.704072 27455 log.cc:824] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:28.704069 27453 log.cc:824] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:28.704574 27452 raft_consensus.cc:348] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.704746 27452 raft_consensus.cc:374] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:28.704761 27453 tablet_bootstrap.cc:492] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: No bootstrap required, opened a new log
I20260925 15:19:28.704792 27452 raft_consensus.cc:729] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1e66f02adf4624928da3a604a972bd, State: Initialized, Role: FOLLOWER
I20260925 15:19:28.704818 27453 ts_tablet_manager.cc:1397] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:28.704905 27452 consensus_queue.cc:260] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.705056 27455 tablet_bootstrap.cc:492] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: No bootstrap required, opened a new log
I20260925 15:19:28.705117 27455 ts_tablet_manager.cc:1397] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:28.705170 27452 ts_tablet_manager.cc:1428] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:28.705288 27176 heartbeater.cc:503] Master 127.8.43.190:41137 was elected leader, sending a full tablet report...
I20260925 15:19:28.707439 27453 raft_consensus.cc:348] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.707607 27453 raft_consensus.cc:374] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:28.707641 27453 raft_consensus.cc:729] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75765063ff04d70a759e5725b2c3cb8, State: Initialized, Role: FOLLOWER
I20260925 15:19:28.707574 27455 raft_consensus.cc:348] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.707708 27455 raft_consensus.cc:374] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:28.707749 27455 raft_consensus.cc:729] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754dd5bfbf764847acea0755364176f1, State: Initialized, Role: FOLLOWER
I20260925 15:19:28.707767 27453 consensus_queue.cc:260] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.707849 27455 consensus_queue.cc:260] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.708014 27453 ts_tablet_manager.cc:1428] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:28.708081 27455 ts_tablet_manager.cc:1428] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:28.708083 27438 heartbeater.cc:503] Master 127.8.43.190:41137 was elected leader, sending a full tablet report...
I20260925 15:19:28.708209 27307 heartbeater.cc:503] Master 127.8.43.190:41137 was elected leader, sending a full tablet report...
I20260925 15:19:28.769234 27459 raft_consensus.cc:482] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:28.769356 27459 raft_consensus.cc:504] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.769784 27459 leader_election.cc:288] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754dd5bfbf764847acea0755364176f1 (127.8.43.130:32929), db1e66f02adf4624928da3a604a972bd (127.8.43.129:46389)
I20260925 15:19:28.773762 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.770061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:58241 (local address 127.8.43.129:46389)
0925 15:19:28.770133 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.770136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.770147 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:28.770178 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.770180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.770193 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.770257 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:28.770486 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.771305 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.772784 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.773331 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.773415 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.773424 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.773649 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.773693 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:19:28.773762 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.770075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:50829 (local address 127.8.43.130:32929)
0925 15:19:28.770140 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.770142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.770450 (+ 308us) server_negotiation.cc:407] Connection header received
0925 15:19:28.770487 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.770488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.770498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.770569 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:28.770707 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.771426 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.772784 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.773331 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.773417 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.773424 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.773656 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.773693 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:19:28.773768 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.769961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46389 (local address 127.8.43.131:58241)
0925 15:19:28.770022 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.770037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.770095 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.770277 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.770278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.770281 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:28.770356 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.770358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.771407 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.771408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.772689 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.772693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.773455 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.773456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.773544 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.773548 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.773617 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.773690 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":30}
I20260925 15:19:28.773779 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.769971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:32929 (local address 127.8.43.131:50829)
0925 15:19:28.770303 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.770313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:28.770382 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:28.770580 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:28.770581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:28.770582 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:28.770633 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.770635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.771511 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.771512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.772693 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:28.772695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.773453 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.773454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:28.773544 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.773549 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:19:28.773617 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:19:28.773678 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":155,"thread_start_us":115,"threads_started":1}
I20260925 15:19:28.774194 27130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b562176a6d4fdfb39137c4bf0ac231" candidate_uuid: "b75765063ff04d70a759e5725b2c3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1e66f02adf4624928da3a604a972bd" is_pre_election: true
I20260925 15:19:28.774394 27130 raft_consensus.cc:2454] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75765063ff04d70a759e5725b2c3cb8 in term 0.
I20260925 15:19:28.774516 27261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b562176a6d4fdfb39137c4bf0ac231" candidate_uuid: "b75765063ff04d70a759e5725b2c3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754dd5bfbf764847acea0755364176f1" is_pre_election: true
I20260925 15:19:28.774667 27261 raft_consensus.cc:2454] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75765063ff04d70a759e5725b2c3cb8 in term 0.
I20260925 15:19:28.774667 27328 leader_election.cc:302] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b75765063ff04d70a759e5725b2c3cb8, db1e66f02adf4624928da3a604a972bd; no voters:
I20260925 15:19:28.774791 27459 raft_consensus.cc:2781] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:28.774866 27459 raft_consensus.cc:482] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:28.774891 27459 raft_consensus.cc:3037] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:28.775684 27459 raft_consensus.cc:504] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.775826 27459 leader_election.cc:288] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [CANDIDATE]: Term 1 election: Requested vote from peers 754dd5bfbf764847acea0755364176f1 (127.8.43.130:32929), db1e66f02adf4624928da3a604a972bd (127.8.43.129:46389)
I20260925 15:19:28.775952 27261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b562176a6d4fdfb39137c4bf0ac231" candidate_uuid: "b75765063ff04d70a759e5725b2c3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754dd5bfbf764847acea0755364176f1"
I20260925 15:19:28.775997 27130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b562176a6d4fdfb39137c4bf0ac231" candidate_uuid: "b75765063ff04d70a759e5725b2c3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1e66f02adf4624928da3a604a972bd"
I20260925 15:19:28.776043 27261 raft_consensus.cc:3037] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:28.776105 27130 raft_consensus.cc:3037] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:28.776861 27261 raft_consensus.cc:2454] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75765063ff04d70a759e5725b2c3cb8 in term 1.
I20260925 15:19:28.776868 27130 raft_consensus.cc:2454] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75765063ff04d70a759e5725b2c3cb8 in term 1.
I20260925 15:19:28.777031 27328 leader_election.cc:302] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b75765063ff04d70a759e5725b2c3cb8, db1e66f02adf4624928da3a604a972bd; no voters:
I20260925 15:19:28.777140 27459 raft_consensus.cc:2781] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:28.777289 27459 raft_consensus.cc:686] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 1 LEADER]: Becoming Leader. State: Replica: b75765063ff04d70a759e5725b2c3cb8, State: Running, Role: LEADER
I20260925 15:19:28.777395 27459 consensus_queue.cc:237] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } }
I20260925 15:19:28.778306 26986 catalog_manager.cc:5495] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b75765063ff04d70a759e5725b2c3cb8 (127.8.43.131). New cstate: current_term: 1 leader_uuid: "b75765063ff04d70a759e5725b2c3cb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b75765063ff04d70a759e5725b2c3cb8" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 } health_report { overall_health: UNKNOWN } } }
I20260925 15:19:28.791690 27451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.789313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.8.43.131:36885)
0925 15:19:28.789371 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:28.789373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:28.789381 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:19:28.789450 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:28.789451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:28.789481 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:28.789542 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:28.789661 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.790120 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.790837 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:28.791057 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:28.791111 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:28.791366 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:28.791452 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:28.791538 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:28.791606 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:19:28.791613 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:19:28.793606 27130 raft_consensus.cc:1261] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 1 FOLLOWER]: Refusing update from remote peer b75765063ff04d70a759e5725b2c3cb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:28.793623 27261 raft_consensus.cc:1261] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 1 FOLLOWER]: Refusing update from remote peer b75765063ff04d70a759e5725b2c3cb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:28.794037 27462 consensus_queue.cc:1035] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1e66f02adf4624928da3a604a972bd" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:28.794201 27459 consensus_queue.cc:1035] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754dd5bfbf764847acea0755364176f1" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:28.796624 27465 mvcc.cc:205] Tried to move back new op lower bound from 7333271833775611904 to 7333271833713176576. Current Snapshot: MvccSnapshot[applied={T|T < 7333271833775611904}]
I20260925 15:19:28.796872 27467 mvcc.cc:205] Tried to move back new op lower bound from 7333271833775611904 to 7333271833713176576. Current Snapshot: MvccSnapshot[applied={T|T < 7333271833775611904}]
I20260925 15:19:28.797389 27466 mvcc.cc:205] Tried to move back new op lower bound from 7333271833775611904 to 7333271833713176576. Current Snapshot: MvccSnapshot[applied={T|T < 7333271833775611904}]
I20260925 15:19:28.798848 26987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:28.798939 26987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:28.799777 26987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e05bbb43828546518de71e6a09f71dab: Sending DeleteTablet for 3 replicas of tablet 32b562176a6d4fdfb39137c4bf0ac231
I20260925 15:19:28.800124 27241 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b562176a6d4fdfb39137c4bf0ac231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:28 UTC) from {username='slave'} at 127.0.0.1:34784
I20260925 15:19:28.800170 27110 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b562176a6d4fdfb39137c4bf0ac231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:28 UTC) from {username='slave'} at 127.0.0.1:47326
I20260925 15:19:28.800266 27372 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b562176a6d4fdfb39137c4bf0ac231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:28 UTC) from {username='slave'} at 127.0.0.1:32774
I20260925 15:19:28.800370 27475 tablet_replica.cc:330] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: stopping tablet replica
I20260925 15:19:28.800469 27475 raft_consensus.cc:2229] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:28.800482 27477 tablet_replica.cc:330] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: stopping tablet replica
I20260925 15:19:28.800570 27475 raft_consensus.cc:2258] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:28.800567 27477 raft_consensus.cc:2229] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:28.800649 27477 raft_consensus.cc:2258] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:28.800879 27476 tablet_replica.cc:330] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd: stopping tablet replica
I20260925 15:19:28.800949 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 27048
I20260925 15:19:28.801059 27476 raft_consensus.cc:2229] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:28.801201 27476 raft_consensus.cc:2258] T 32b562176a6d4fdfb39137c4bf0ac231 P db1e66f02adf4624928da3a604a972bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:28.803066 27477 ts_tablet_manager.cc:1905] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:28.804016 27475 ts_tablet_manager.cc:1905] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:28.805083 27477 ts_tablet_manager.cc:1918] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:28.805161 27477 log.cc:1196] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/32b562176a6d4fdfb39137c4bf0ac231
I20260925 15:19:28.805423 27477 ts_tablet_manager.cc:1939] T 32b562176a6d4fdfb39137c4bf0ac231 P b75765063ff04d70a759e5725b2c3cb8: Deleting consensus metadata
I20260925 15:19:28.805796 26973 catalog_manager.cc:4841] TS b75765063ff04d70a759e5725b2c3cb8 (127.8.43.131:36885): tablet 32b562176a6d4fdfb39137c4bf0ac231 (table test-table [id=4a87f9885b49418994aa74f30c932235]) successfully deleted
W20260925 15:19:28.806422 26975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:46389 (error 108)
I20260925 15:19:28.806643 27475 ts_tablet_manager.cc:1918] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:28.806721 27475 log.cc:1196] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/32b562176a6d4fdfb39137c4bf0ac231
I20260925 15:19:28.806990 27475 ts_tablet_manager.cc:1939] T 32b562176a6d4fdfb39137c4bf0ac231 P 754dd5bfbf764847acea0755364176f1: Deleting consensus metadata
I20260925 15:19:28.807446 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 27179
I20260925 15:19:28.807466 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.807342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:46389 (local address 127.0.0.1:47338)
0925 15:19:28.807391 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.807430 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 15:19:28.808173 26975 catalog_manager.cc:4584] TS db1e66f02adf4624928da3a604a972bd (127.8.43.129:46389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32b562176a6d4fdfb39137c4bf0ac231: Network error: Client connection negotiation failed: client connection to 127.8.43.129:46389: connect: Connection refused (error 111)
I20260925 15:19:28.812662 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:28.812546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:32929 (local address 127.0.0.1:34794)
0925 15:19:28.812604 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:28.812633 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.130:32929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:19:28.812749 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 27310
I20260925 15:19:28.817030 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 26957
2026-09-25T15:19:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (712 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:28 dist-test-slave-c5d6 krb5kdc[27481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:28 dist-test-slave-c5d6 krb5kdc[27481](info): set up 2 sockets
Sep 25 15:19:28 dist-test-slave-c5d6 krb5kdc[27481](info): commencing operation
krb5kdc: starting...
W20260925 15:19:29.862905 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:29 dist-test-slave-c5d6 krb5kdc[27481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:29.982992 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44397
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:30.064478 27494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:30.064675 27494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:30.064697 27494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:30.065755 27494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:30.065788 27494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:30.065802 27494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:30.065814 27494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:30.065826 27494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:30.067323 27494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:30.067512 27494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:30.067663 27494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:30.072302 27499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:30.072309 27500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:30.072670 27494 server_base.cc:1034] running on GCE node
W20260925 15:19:30.072322 27502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:30.072956 27494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:30.073276 27494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:30.074438 27494 hybrid_clock.cc:648] HybridClock initialized: now 1790349570074434 us; error 34 us; skew 500 ppm
Sep 25 15:19:30 dist-test-slave-c5d6 krb5kdc[27481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:30.077021 27494 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:30.077453 27494 webserver.cc:460] Webserver started at http://127.8.43.190:46003/ using document root <none> and password file <none>
I20260925 15:19:30.077689 27494 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:30.077756 27494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:30.077905 27494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:30.078969 27494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "5bcb7abd5e8740deb562b01f29876d05"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "c4f77e65433c8df06c98928e83c03eb0"
server_key_iv: "139ac64b86cda9a74caa1a7837837961"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.079357 27494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "5bcb7abd5e8740deb562b01f29876d05"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "c4f77e65433c8df06c98928e83c03eb0"
server_key_iv: "139ac64b86cda9a74caa1a7837837961"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.080636 27494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:30.081328 27509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.081516 27494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:30.081610 27494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "5bcb7abd5e8740deb562b01f29876d05"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "c4f77e65433c8df06c98928e83c03eb0"
server_key_iv: "139ac64b86cda9a74caa1a7837837961"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.081689 27494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:30.103488 27494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:30.105377 27494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:30.105544 27494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:30.109419 27494 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44397
I20260925 15:19:30.109480 27561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44397 every 8 connection(s)
I20260925 15:19:30.109823 27494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:30.110694 27562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:30.113059 27562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05: Bootstrap starting.
I20260925 15:19:30.113626 27562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:30.113909 27562 log.cc:824] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:30.114622 27562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05: No bootstrap required, opened a new log
I20260925 15:19:30.116235 27562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcb7abd5e8740deb562b01f29876d05" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44397 } }
I20260925 15:19:30.116372 27562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:30.116395 27562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bcb7abd5e8740deb562b01f29876d05, State: Initialized, Role: FOLLOWER
I20260925 15:19:30.116567 27562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcb7abd5e8740deb562b01f29876d05" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44397 } }
I20260925 15:19:30.116631 27562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:30.116652 27562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:30.116725 27562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:30.117234 27562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcb7abd5e8740deb562b01f29876d05" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44397 } }
I20260925 15:19:30.117339 27562 leader_election.cc:302] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bcb7abd5e8740deb562b01f29876d05; no voters:
I20260925 15:19:30.117515 27562 leader_election.cc:288] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:30.117555 27565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:30.117730 27565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [term 1 LEADER]: Becoming Leader. State: Replica: 5bcb7abd5e8740deb562b01f29876d05, State: Running, Role: LEADER
I20260925 15:19:30.117820 27565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcb7abd5e8740deb562b01f29876d05" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44397 } }
I20260925 15:19:30.117882 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 27494
I20260925 15:19:30.117995 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:30.118234 8366 external_mini_cluster.cc:1428] Setting key eedd544f6916a7da46b2b8a4a9ea149a
I20260925 15:19:30.118196 27566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bcb7abd5e8740deb562b01f29876d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcb7abd5e8740deb562b01f29876d05" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44397 } } }
I20260925 15:19:30.118278 27566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:30.118337 27567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bcb7abd5e8740deb562b01f29876d05. Latest consensus state: current_term: 1 leader_uuid: "5bcb7abd5e8740deb562b01f29876d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bcb7abd5e8740deb562b01f29876d05" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44397 } } }
I20260925 15:19:30.118408 27567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:30.118597 27562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:30.119351 27569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:30.119899 27569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:30.121670 27569 catalog_manager.cc:1347] Generated new cluster ID: 3a95b9fc426a4f54ad10ed0a2eed3565
I20260925 15:19:30.121719 27569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:19:30 dist-test-slave-c5d6 krb5kdc[27481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:30.129604 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.118716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.8.43.190:44397)
0925 15:19:30.119050 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.119059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.119114 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:19:30.119276 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.119283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.119342 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.119518 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:30.120283 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.121624 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.122500 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.122923 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.123055 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.125553 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:30.125558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:30.125558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:30.125571 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:30.127731 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.128127 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.128130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.128130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.128244 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.128472 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.128474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.128474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.128788 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:30.128956 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.129196 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.129445 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":85,"threads_started":1}
I20260925 15:19:30.135978 27569 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:30.136644 27569 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:30.140609 27569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05: Generated new TSK 0
I20260925 15:19:30.140796 27569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.43.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
Entry for principal kudu/127.8.43.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab.
I20260925 15:19:30.169977 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44397
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:30.259769 27588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:30.259974 27588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:30.260000 27588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:30.261157 27588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:30.261196 27588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:30.261222 27588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:19:30.262662 27588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.129.keytab
--principal=kudu/127.8.43.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:30.262899 27588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:30.263151 27588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:30.268074 27593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:30.268054 27596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:30.268062 27594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:30.268496 27588 server_base.cc:1034] running on GCE node
I20260925 15:19:30.268736 27588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:30.269001 27588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:30.270184 27588 hybrid_clock.cc:648] HybridClock initialized: now 1790349570270168 us; error 33 us; skew 500 ppm
Sep 25 15:19:30 dist-test-slave-c5d6 krb5kdc[27481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:30.272825 27588 init.cc:377] Logged in from keytab as kudu/127.8.43.129@KRBTEST.COM (short username kudu)
I20260925 15:19:30.273298 27588 webserver.cc:460] Webserver started at http://127.8.43.129:36893/ using document root <none> and password file <none>
I20260925 15:19:30.273545 27588 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:30.273614 27588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:30.273783 27588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:30.274887 27588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "e9fb79b8f9aa42438699e67bfe69d6c9"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "505500aa7f09f5230268fc5be383a9c5"
server_key_iv: "a92000b67c54fc68a443d45a943739fe"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.275331 27588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "e9fb79b8f9aa42438699e67bfe69d6c9"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "505500aa7f09f5230268fc5be383a9c5"
server_key_iv: "a92000b67c54fc68a443d45a943739fe"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.276772 27588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:30.277549 27603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.277738 27588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.277840 27588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "e9fb79b8f9aa42438699e67bfe69d6c9"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "505500aa7f09f5230268fc5be383a9c5"
server_key_iv: "a92000b67c54fc68a443d45a943739fe"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.277927 27588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:30.288592 27588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:30.290509 27588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:30.290675 27588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:30.290905 27588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:30.291286 27588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:30.291345 27588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.291379 27588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:30.291410 27588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.297339 27588 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:44853
I20260925 15:19:30.297389 27716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:44853 every 8 connection(s)
I20260925 15:19:30.297732 27588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:19:30 dist-test-slave-c5d6 krb5kdc[27481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.129@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:30.305109 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 27588
I20260925 15:19:30.305224 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:30.305435 8366 external_mini_cluster.cc:1428] Setting key 7a7f2a805523df092842d671c9a983ef
I20260925 15:19:30.307618 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.298487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:57807 (local address 127.8.43.190:44397)
0925 15:19:30.298555 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.298557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.298743 (+ 186us) server_negotiation.cc:407] Connection header received
0925 15:19:30.299202 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.299203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.299250 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.299314 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:30.300215 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.300719 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.301643 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.301835 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.301891 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.304011 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:30.304012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:30.304013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:30.304020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:30.306123 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.306552 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.306553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.306554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.306616 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.306870 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.306871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.306871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.307016 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:30.307094 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.307431 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.307527 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:19:30.307828 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.298439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44397 (local address 127.8.43.129:57807)
0925 15:19:30.298601 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.298643 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:30.299122 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:30.299392 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:30.299396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:30.299943 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:30.300133 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.300137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.300796 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.300797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.301574 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.301576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.301903 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.301904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.302008 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.302435 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:30.302442 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:30.303883 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:30.306205 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:30.306212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:30.306214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:30.306439 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:30.306686 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:30.306687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:30.306687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:30.306794 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:30.307086 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:30.307093 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:30.307279 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 15:19:30.307407 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 15:19:30.307578 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260925 15:19:30.308398 27717 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44397
I20260925 15:19:30.308503 27717 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:30.308694 27717 heartbeater.cc:511] Master 127.8.43.190:44397 requested a full tablet report, sending...
I20260925 15:19:30.309224 27526 ts_manager.cc:194] Registered new tserver with Master: e9fb79b8f9aa42438699e67bfe69d6c9 (127.8.43.129:44853)
I20260925 15:19:30.309906 27526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.129@KRBTEST.COM'} at 127.8.43.129:57807
WARNING: no policy specified for kudu/127.8.43.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
Entry for principal kudu/127.8.43.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab.
I20260925 15:19:30.330052 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44397
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:30.411136 27722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:30.411450 27722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:30.411492 27722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:30.412647 27722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:30.412716 27722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:30.412761 27722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:19:30.414196 27722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.130.keytab
--principal=kudu/127.8.43.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:30.414462 27722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:30.414696 27722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:30.419785 27730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:30.419790 27728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:30.419797 27727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:30.420060 27722 server_base.cc:1034] running on GCE node
I20260925 15:19:30.420290 27722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:30.420531 27722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:30.421696 27722 hybrid_clock.cc:648] HybridClock initialized: now 1790349570421684 us; error 41 us; skew 500 ppm
Sep 25 15:19:30 dist-test-slave-c5d6 krb5kdc[27481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:30.424307 27722 init.cc:377] Logged in from keytab as kudu/127.8.43.130@KRBTEST.COM (short username kudu)
I20260925 15:19:30.424741 27722 webserver.cc:460] Webserver started at http://127.8.43.130:35117/ using document root <none> and password file <none>
I20260925 15:19:30.424966 27722 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:30.425038 27722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:30.425181 27722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:30.426244 27722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "f98144279e5343ab8ce62f85508c7b3c"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "408d3007341cbfee4cdc8aafa250b5d3"
server_key_iv: "e4b45115b77b5fe625b6718b1798ad8b"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.426609 27722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "f98144279e5343ab8ce62f85508c7b3c"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "408d3007341cbfee4cdc8aafa250b5d3"
server_key_iv: "e4b45115b77b5fe625b6718b1798ad8b"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.428048 27722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:30.428824 27737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.429034 27722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:30.429133 27722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "f98144279e5343ab8ce62f85508c7b3c"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "408d3007341cbfee4cdc8aafa250b5d3"
server_key_iv: "e4b45115b77b5fe625b6718b1798ad8b"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.429214 27722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:30.447396 27722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:30.449728 27722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:30.449939 27722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:30.450201 27722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:30.450525 27722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:30.450582 27722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.450621 27722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:30.450671 27722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.457104 27722 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:45451
I20260925 15:19:30.457145 27850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:45451 every 8 connection(s)
I20260925 15:19:30.457489 27722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:19:30 dist-test-slave-c5d6 krb5kdc[27481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.130@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:30.464594 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 27722
I20260925 15:19:30.464717 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:30.464941 8366 external_mini_cluster.cc:1428] Setting key 6aa71a2d1e3695c466f6a085887a9ff9
I20260925 15:19:30.467326 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.458284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:47023 (local address 127.8.43.190:44397)
0925 15:19:30.458355 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.458358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.458607 (+ 249us) server_negotiation.cc:407] Connection header received
0925 15:19:30.459110 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.459111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.459122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.459181 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:30.460154 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.460658 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.461661 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.461849 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.461906 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.464042 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:30.464043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:30.464044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:30.464050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:30.465730 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.466260 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.466261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.466261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.466316 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.466617 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.466618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.466618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.466720 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:30.466772 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.467114 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.467178 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:19:30.467551 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.458269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44397 (local address 127.8.43.130:47023)
0925 15:19:30.458445 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.458490 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:30.459015 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:30.459255 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:30.459258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:30.459831 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:30.460046 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.460051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.460745 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.460746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.461572 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.461574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.461950 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.461950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.462051 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.462505 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:30.462513 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:30.463937 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:30.465808 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:30.465816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:30.465819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:30.466167 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:30.466391 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:30.466392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:30.466393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:30.466546 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:30.466787 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:30.466793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:30.466972 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 15:19:30.467098 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:19:30.467315 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260925 15:19:30.468056 27851 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44397
I20260925 15:19:30.468168 27851 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:30.468359 27851 heartbeater.cc:511] Master 127.8.43.190:44397 requested a full tablet report, sending...
I20260925 15:19:30.468794 27526 ts_manager.cc:194] Registered new tserver with Master: f98144279e5343ab8ce62f85508c7b3c (127.8.43.130:45451)
I20260925 15:19:30.469172 27526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.130@KRBTEST.COM'} at 127.8.43.130:47023
WARNING: no policy specified for kudu/127.8.43.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
Entry for principal kudu/127.8.43.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab.
I20260925 15:19:30.494324 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44397
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:30.574209 27856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:30.574401 27856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:30.574424 27856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:30.575627 27856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:30.575668 27856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:30.575695 27856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:19:30.577122 27856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.131.keytab
--principal=kudu/127.8.43.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:30.577302 27856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:30.577472 27856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:30.582300 27862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:30.582309 27864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:30.582332 27861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:30.582538 27856 server_base.cc:1034] running on GCE node
I20260925 15:19:30.582782 27856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:30.582973 27856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:30.584110 27856 hybrid_clock.cc:648] HybridClock initialized: now 1790349570584099 us; error 25 us; skew 500 ppm
Sep 25 15:19:30 dist-test-slave-c5d6 krb5kdc[27481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:30.586565 27856 init.cc:377] Logged in from keytab as kudu/127.8.43.131@KRBTEST.COM (short username kudu)
I20260925 15:19:30.586944 27856 webserver.cc:460] Webserver started at http://127.8.43.131:41173/ using document root <none> and password file <none>
I20260925 15:19:30.587122 27856 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:30.587162 27856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:30.587309 27856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:30.588261 27856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "b131bbe7b7d841bd9003bc901e783614"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "55aa584e22835f6f80506f1881e9c2b2"
server_key_iv: "bff6726a64d59847821c5035e24a8392"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.588584 27856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "b131bbe7b7d841bd9003bc901e783614"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "55aa584e22835f6f80506f1881e9c2b2"
server_key_iv: "bff6726a64d59847821c5035e24a8392"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.589879 27856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:30.590615 27871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.590777 27856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:30.590867 27856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "b131bbe7b7d841bd9003bc901e783614"
format_stamp: "Formatted at 2026-09-25 15:19:30 on dist-test-slave-c5d6"
server_key: "55aa584e22835f6f80506f1881e9c2b2"
server_key_iv: "bff6726a64d59847821c5035e24a8392"
server_key_version: "encryptionkey@0"
I20260925 15:19:30.590919 27856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:30.603645 27856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:30.605436 27856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:30.605542 27856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:30.605734 27856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:30.606057 27856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:30.606091 27856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.606112 27856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:30.606158 27856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:30.611776 27856 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:34651
I20260925 15:19:30.611847 27984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:34651 every 8 connection(s)
I20260925 15:19:30.612162 27856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:19:30 dist-test-slave-c5d6 krb5kdc[27481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349570, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.131@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:30.619163 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 27856
I20260925 15:19:30.619304 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:30.619557 8366 external_mini_cluster.cc:1428] Setting key 7f80726408a97545aa7a4532abc3e898
I20260925 15:19:30.621403 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.612882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:40661 (local address 127.8.43.190:44397)
0925 15:19:30.612954 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.612956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.613139 (+ 183us) server_negotiation.cc:407] Connection header received
0925 15:19:30.613583 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.613584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.613593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.613648 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:30.614536 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.615025 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.616083 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.616265 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.616316 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.618321 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:30.618323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:30.618323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:30.618330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:30.619999 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.620450 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.620451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.620452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.620512 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.620777 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.620778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.620778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.620891 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:30.620965 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.621234 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.621314 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:19:30.621596 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.612846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:44397 (local address 127.8.43.131:40661)
0925 15:19:30.613008 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.613048 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:30.613504 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:30.613696 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:30.613699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:30.614222 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:30.614447 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.614454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.615105 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.615106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.615943 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.615945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.616336 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.616337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.616435 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.616857 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:30.616865 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:30.618172 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:30.620087 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:30.620093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:30.620095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:30.620382 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:30.620572 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:30.620573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:30.620574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:30.620697 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:30.620958 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:30.620965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:30.621102 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:19:30.621213 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:19:30.621362 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260925 15:19:30.622052 27985 heartbeater.cc:347] Connected to a master server at 127.8.43.190:44397
I20260925 15:19:30.622128 27985 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:30.622311 27985 heartbeater.cc:511] Master 127.8.43.190:44397 requested a full tablet report, sending...
I20260925 15:19:30.622658 27526 ts_manager.cc:194] Registered new tserver with Master: b131bbe7b7d841bd9003bc901e783614 (127.8.43.131:34651)
I20260925 15:19:30.622999 27526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.43.131@KRBTEST.COM'} at 127.8.43.131:40661
I20260925 15:19:30.631471 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:30.638800 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.632910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.8.43.190:44397)
0925 15:19:30.633019 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.633021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.633120 (+ 99us) server_negotiation.cc:407] Connection header received
0925 15:19:30.633157 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.633157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.633167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.633215 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:30.633710 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.634232 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.634929 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.635119 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.635168 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.636337 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:30.636339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:30.636339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:30.636345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:30.637826 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.638120 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.638121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.638121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.638171 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:30.638332 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:30.638332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:30.638333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:30.638463 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:30.638553 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.638636 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.638703 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:19:30.640210 27526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:30.642043 27526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:30.648538 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.644393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:45451 (local address 127.0.0.1:42746)
0925 15:19:30.644577 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.644614 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:30.644701 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:30.645045 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:30.645049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:30.645051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:30.645192 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.645195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.646103 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.646104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.647281 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.647283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.648192 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.648193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.648272 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.648287 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:19:30.648373 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:19:30.648424 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":114,"threads_started":1}
I20260925 15:19:30.648715 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.644764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.8.43.131:34651)
0925 15:19:30.644929 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.644933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.645054 (+ 121us) server_negotiation.cc:407] Connection header received
0925 15:19:30.645097 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.645099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.645114 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.645183 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:30.645377 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.646453 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.647739 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.648310 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.648380 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.648447 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.648559 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.648607 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260925 15:19:30.648839 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.644586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.8.43.130:45451)
0925 15:19:30.644766 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.644771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.644781 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:19:30.644826 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.644828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.644845 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.644919 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:30.645272 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.646001 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.647361 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.648103 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.648178 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.648232 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.648727 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.648769 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260925 15:19:30.648659 27997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.644394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:34651 (local address 127.0.0.1:50030)
0925 15:19:30.644923 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.644932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:30.644993 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:30.645263 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:30.645264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:30.645266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:30.645297 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.645299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.646554 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.646555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.647616 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.647619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.648392 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.648393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.648456 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.648461 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:19:30.648535 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:19:30.648576 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":4,"thread_start_us":115,"threads_started":1}
I20260925 15:19:30.649292 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.644394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44853 (local address 127.0.0.1:38130)
0925 15:19:30.644798 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.644806 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:30.644840 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:30.645113 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:30.645114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:30.645116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:30.645167 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.645169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.646503 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.646504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.647640 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.647643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.648799 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.648800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.648863 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.648867 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:30.649167 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 15:19:30.649201 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":42,"threads_started":1}
I20260925 15:19:30.649526 28000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.644632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.8.43.129:44853)
0925 15:19:30.644826 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.644831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.644844 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:19:30.644916 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.644919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.644936 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.645023 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:30.645248 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.646394 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.647770 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.648493 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.648554 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.648606 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.649386 (+ 780us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.649445 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260925 15:19:30.649588 27785 tablet_service.cc:1467] Processing CreateTablet for tablet 12450c74ef9b41e182a398bfcc116286 (DEFAULT_TABLE table=test-table [id=d76d37b20789491b8c02896035bb6e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:30.649979 27785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12450c74ef9b41e182a398bfcc116286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:30.650151 27651 tablet_service.cc:1467] Processing CreateTablet for tablet 12450c74ef9b41e182a398bfcc116286 (DEFAULT_TABLE table=test-table [id=d76d37b20789491b8c02896035bb6e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:30.650449 27651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12450c74ef9b41e182a398bfcc116286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:30.652841 28002 tablet_bootstrap.cc:492] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Bootstrap starting.
I20260925 15:19:30.653206 28003 tablet_bootstrap.cc:492] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9: Bootstrap starting.
I20260925 15:19:30.653257 27919 tablet_service.cc:1467] Processing CreateTablet for tablet 12450c74ef9b41e182a398bfcc116286 (DEFAULT_TABLE table=test-table [id=d76d37b20789491b8c02896035bb6e81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:30.653569 27919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12450c74ef9b41e182a398bfcc116286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:30.653622 28002 tablet_bootstrap.cc:654] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:30.653852 28003 tablet_bootstrap.cc:654] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:30.654160 28002 log.cc:824] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:30.654417 28003 log.cc:824] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:30.654968 28002 tablet_bootstrap.cc:492] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: No bootstrap required, opened a new log
I20260925 15:19:30.655048 28002 ts_tablet_manager.cc:1397] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:30.656894 28006 tablet_bootstrap.cc:492] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Bootstrap starting.
I20260925 15:19:30.657188 28002 raft_consensus.cc:348] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.657382 28002 raft_consensus.cc:374] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:30.657434 28002 raft_consensus.cc:729] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98144279e5343ab8ce62f85508c7b3c, State: Initialized, Role: FOLLOWER
I20260925 15:19:30.657544 28003 tablet_bootstrap.cc:492] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9: No bootstrap required, opened a new log
I20260925 15:19:30.657575 28006 tablet_bootstrap.cc:654] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:30.657565 28002 consensus_queue.cc:260] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.657617 28003 ts_tablet_manager.cc:1397] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 15:19:30.657848 27851 heartbeater.cc:503] Master 127.8.43.190:44397 was elected leader, sending a full tablet report...
I20260925 15:19:30.657848 28002 ts_tablet_manager.cc:1428] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:30.658135 28006 log.cc:824] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:30.658876 28006 tablet_bootstrap.cc:492] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: No bootstrap required, opened a new log
I20260925 15:19:30.659010 28006 ts_tablet_manager.cc:1397] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:19:30.659664 28003 raft_consensus.cc:348] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.659801 28003 raft_consensus.cc:374] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:30.659822 28003 raft_consensus.cc:729] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fb79b8f9aa42438699e67bfe69d6c9, State: Initialized, Role: FOLLOWER
I20260925 15:19:30.659982 28003 consensus_queue.cc:260] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.660341 28003 ts_tablet_manager.cc:1428] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:19:30.660353 27717 heartbeater.cc:503] Master 127.8.43.190:44397 was elected leader, sending a full tablet report...
I20260925 15:19:30.660837 28006 raft_consensus.cc:348] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.661015 28006 raft_consensus.cc:374] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:30.661065 28006 raft_consensus.cc:729] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b131bbe7b7d841bd9003bc901e783614, State: Initialized, Role: FOLLOWER
I20260925 15:19:30.661185 28006 consensus_queue.cc:260] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.661456 28006 ts_tablet_manager.cc:1428] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:30.661481 27985 heartbeater.cc:503] Master 127.8.43.190:44397 was elected leader, sending a full tablet report...
W20260925 15:19:30.708093 27852 tablet.cc:2369] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:19:30.798492 27718 tablet.cc:2369] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:19:30.862711 27986 tablet.cc:2369] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:30.872756 28009 raft_consensus.cc:482] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:30.872865 28009 raft_consensus.cc:504] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.873190 28009 leader_election.cc:288] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b131bbe7b7d841bd9003bc901e783614 (127.8.43.131:34651), f98144279e5343ab8ce62f85508c7b3c (127.8.43.130:45451)
I20260925 15:19:30.876302 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.873417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:57669 (local address 127.8.43.131:34651)
0925 15:19:30.873480 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.873482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.873571 (+ 89us) server_negotiation.cc:407] Connection header received
0925 15:19:30.873664 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.873665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.873674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.873747 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:30.873896 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.874398 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.875552 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.875932 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.876031 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.876039 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.876187 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.876223 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:19:30.876314 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.873447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:45917 (local address 127.8.43.130:45451)
0925 15:19:30.873494 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.873497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.873577 (+ 80us) server_negotiation.cc:407] Connection header received
0925 15:19:30.873659 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.873660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.873671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.873726 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:30.873905 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.874396 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.875586 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.875954 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.876046 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.876054 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.876205 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.876244 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:19:30.876343 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.873354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:45451 (local address 127.8.43.129:45917)
0925 15:19:30.873490 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.873503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:30.873590 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:30.873739 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:30.873740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:30.873742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:30.873833 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.873835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.874494 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.874495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.875491 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.875493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.876043 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.876045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.876116 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.876119 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:30.876199 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:19:30.876240 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":31,"thread_start_us":70,"threads_started":1}
I20260925 15:19:30.876344 28012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.873335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:34651 (local address 127.8.43.129:57669)
0925 15:19:30.873493 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.873503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:30.873588 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:30.873750 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:30.873751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:30.873753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:30.873826 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.873828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.874495 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.874496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.875441 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:30.875444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.876031 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.876032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:30.876103 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.876107 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:19:30.876178 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:19:30.876223 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":33,"thread_start_us":85,"threads_started":1}
I20260925 15:19:30.876711 27805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12450c74ef9b41e182a398bfcc116286" candidate_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98144279e5343ab8ce62f85508c7b3c" is_pre_election: true
I20260925 15:19:30.876873 27805 raft_consensus.cc:2454] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fb79b8f9aa42438699e67bfe69d6c9 in term 0.
I20260925 15:19:30.877027 27939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12450c74ef9b41e182a398bfcc116286" candidate_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b131bbe7b7d841bd9003bc901e783614" is_pre_election: true
I20260925 15:19:30.877162 27605 leader_election.cc:302] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9fb79b8f9aa42438699e67bfe69d6c9, f98144279e5343ab8ce62f85508c7b3c; no voters:
I20260925 15:19:30.877190 27939 raft_consensus.cc:2454] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fb79b8f9aa42438699e67bfe69d6c9 in term 0.
I20260925 15:19:30.877290 28009 raft_consensus.cc:2781] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:30.877349 28009 raft_consensus.cc:482] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:30.877372 28009 raft_consensus.cc:3037] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:30.878067 28009 raft_consensus.cc:504] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.878201 28009 leader_election.cc:288] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [CANDIDATE]: Term 1 election: Requested vote from peers b131bbe7b7d841bd9003bc901e783614 (127.8.43.131:34651), f98144279e5343ab8ce62f85508c7b3c (127.8.43.130:45451)
I20260925 15:19:30.878341 27939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12450c74ef9b41e182a398bfcc116286" candidate_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b131bbe7b7d841bd9003bc901e783614"
I20260925 15:19:30.878360 27805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12450c74ef9b41e182a398bfcc116286" candidate_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98144279e5343ab8ce62f85508c7b3c"
I20260925 15:19:30.878417 27939 raft_consensus.cc:3037] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:30.878439 27805 raft_consensus.cc:3037] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:30.879158 27939 raft_consensus.cc:2454] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fb79b8f9aa42438699e67bfe69d6c9 in term 1.
I20260925 15:19:30.879290 27805 raft_consensus.cc:2454] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fb79b8f9aa42438699e67bfe69d6c9 in term 1.
I20260925 15:19:30.879343 27604 leader_election.cc:302] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b131bbe7b7d841bd9003bc901e783614, e9fb79b8f9aa42438699e67bfe69d6c9; no voters:
I20260925 15:19:30.879441 28009 raft_consensus.cc:2781] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:30.879590 28009 raft_consensus.cc:686] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [term 1 LEADER]: Becoming Leader. State: Replica: e9fb79b8f9aa42438699e67bfe69d6c9, State: Running, Role: LEADER
I20260925 15:19:30.879686 28009 consensus_queue.cc:237] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } }
I20260925 15:19:30.880499 27526 catalog_manager.cc:5495] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fb79b8f9aa42438699e67bfe69d6c9 (127.8.43.129). New cstate: current_term: 1 leader_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9fb79b8f9aa42438699e67bfe69d6c9" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 44853 } health_report { overall_health: HEALTHY } } }
I20260925 15:19:30.929697 28000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.927509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.8.43.129:44853)
0925 15:19:30.927582 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:30.927584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:30.927592 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:19:30.927628 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:30.927629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:30.927649 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:30.927710 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:30.927856 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.928321 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.929030 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:30.929213 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:30.929265 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:30.929429 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:30.929527 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:30.929586 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:30.929619 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:19:30.929636 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:19:30.931591 27939 raft_consensus.cc:1261] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 1 FOLLOWER]: Refusing update from remote peer e9fb79b8f9aa42438699e67bfe69d6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:30.931638 27805 raft_consensus.cc:1261] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 1 FOLLOWER]: Refusing update from remote peer e9fb79b8f9aa42438699e67bfe69d6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:30.931958 28009 consensus_queue.cc:1035] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b131bbe7b7d841bd9003bc901e783614" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 34651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:30.932166 28013 consensus_queue.cc:1035] T 12450c74ef9b41e182a398bfcc116286 P e9fb79b8f9aa42438699e67bfe69d6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f98144279e5343ab8ce62f85508c7b3c" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:30.934610 28016 mvcc.cc:205] Tried to move back new op lower bound from 7333271842532954112 to 7333271842323750912. Current Snapshot: MvccSnapshot[applied={T|T < 7333271842532954112}]
I20260925 15:19:30.934834 28017 mvcc.cc:205] Tried to move back new op lower bound from 7333271842532954112 to 7333271842323750912. Current Snapshot: MvccSnapshot[applied={T|T < 7333271842532954112}]
I20260925 15:19:30.935663 28018 mvcc.cc:205] Tried to move back new op lower bound from 7333271842532954112 to 7333271842323750912. Current Snapshot: MvccSnapshot[applied={T|T < 7333271842532954112}]
I20260925 15:19:30.937513 27526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:30.937600 27526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:30.938426 27526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bcb7abd5e8740deb562b01f29876d05: Sending DeleteTablet for 3 replicas of tablet 12450c74ef9b41e182a398bfcc116286
I20260925 15:19:30.938798 27919 tablet_service.cc:1514] Processing DeleteTablet for tablet 12450c74ef9b41e182a398bfcc116286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:30 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:50030
I20260925 15:19:30.938805 27785 tablet_service.cc:1514] Processing DeleteTablet for tablet 12450c74ef9b41e182a398bfcc116286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:30 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:42746
I20260925 15:19:30.939014 27651 tablet_service.cc:1514] Processing DeleteTablet for tablet 12450c74ef9b41e182a398bfcc116286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:30 UTC) from {username='kudu', principal='kudu/127.8.43.190@KRBTEST.COM'} at 127.0.0.1:38130
I20260925 15:19:30.939235 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 27588
I20260925 15:19:30.941054 28026 tablet_replica.cc:330] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: stopping tablet replica
I20260925 15:19:30.941177 28026 raft_consensus.cc:2229] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:30.941294 28026 raft_consensus.cc:2258] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:30.941475 28025 tablet_replica.cc:330] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: stopping tablet replica
I20260925 15:19:30.941573 28025 raft_consensus.cc:2229] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:30.941673 28025 raft_consensus.cc:2258] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:30.942042 28026 ts_tablet_manager.cc:1905] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:30.942092 28025 ts_tablet_manager.cc:1905] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:30.943629 28026 ts_tablet_manager.cc:1918] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:30.943711 28026 log.cc:1196] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/12450c74ef9b41e182a398bfcc116286
I20260925 15:19:30.943955 28026 ts_tablet_manager.cc:1939] T 12450c74ef9b41e182a398bfcc116286 P f98144279e5343ab8ce62f85508c7b3c: Deleting consensus metadata
I20260925 15:19:30.944088 28025 ts_tablet_manager.cc:1918] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:30.944159 28025 log.cc:1196] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/12450c74ef9b41e182a398bfcc116286
I20260925 15:19:30.944353 27511 catalog_manager.cc:4841] TS f98144279e5343ab8ce62f85508c7b3c (127.8.43.130:45451): tablet 12450c74ef9b41e182a398bfcc116286 (table test-table [id=d76d37b20789491b8c02896035bb6e81]) successfully deleted
I20260925 15:19:30.944424 28025 ts_tablet_manager.cc:1939] T 12450c74ef9b41e182a398bfcc116286 P b131bbe7b7d841bd9003bc901e783614: Deleting consensus metadata
I20260925 15:19:30.944810 27510 catalog_manager.cc:4841] TS b131bbe7b7d841bd9003bc901e783614 (127.8.43.131:34651): tablet 12450c74ef9b41e182a398bfcc116286 (table test-table [id=d76d37b20789491b8c02896035bb6e81]) successfully deleted
W20260925 15:19:30.945544 27513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:44853 (error 108)
I20260925 15:19:30.945708 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 27722
I20260925 15:19:30.946743 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:30.946588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:44853 (local address 127.0.0.1:38162)
0925 15:19:30.946652 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:30.946698 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:19:30.946874 27513 catalog_manager.cc:4584] TS e9fb79b8f9aa42438699e67bfe69d6c9 (127.8.43.129:44853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12450c74ef9b41e182a398bfcc116286: Network error: Client connection negotiation failed: client connection to 127.8.43.129:44853: connect: Connection refused (error 111)
I20260925 15:19:30.950650 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 27856
I20260925 15:19:30.955494 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 27494
2026-09-25T15:19:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2138 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:19:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:30Z Disabled control of system clock
I20260925 15:19:30.982657 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38215
--webserver_interface=127.8.43.190
--webserver_port=0
--builtin_ntp_servers=127.8.43.148:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:19:31.061465 28032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:31.061725 28032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:31.061758 28032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:31.062912 28032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:31.062990 28032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:31.063017 28032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:31.063050 28032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:31.063079 28032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:31.064447 28032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:38215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:31.064719 28032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:31.065007 28032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:31.069731 28040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.069731 28037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.069782 28038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:31.069877 28032 server_base.cc:1034] running on GCE node
I20260925 15:19:31.070192 28032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:31.070531 28032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:31.071697 28032 hybrid_clock.cc:648] HybridClock initialized: now 1790349571071681 us; error 43 us; skew 500 ppm
I20260925 15:19:31.072870 28032 webserver.cc:460] Webserver started at http://127.8.43.190:38289/ using document root <none> and password file <none>
I20260925 15:19:31.073163 28032 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:31.073232 28032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:31.073386 28032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:31.074452 28032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "d41bddeff2234910a728902aca6359e6"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "080e5ce95df119410eb893295233de72"
server_key_iv: "3fdd51bf393545ceecacf727930373fc"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.074817 28032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "d41bddeff2234910a728902aca6359e6"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "080e5ce95df119410eb893295233de72"
server_key_iv: "3fdd51bf393545ceecacf727930373fc"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.076208 28032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:19:31.076926 28046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.077116 28032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:31.077221 28032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "d41bddeff2234910a728902aca6359e6"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "080e5ce95df119410eb893295233de72"
server_key_iv: "3fdd51bf393545ceecacf727930373fc"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.077310 28032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:31.100802 28032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:31.101193 28032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:31.101362 28032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:31.105268 28032 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:38215
I20260925 15:19:31.105300 28098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:38215 every 8 connection(s)
I20260925 15:19:31.105659 28032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:31.106657 28099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:31.106772 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 28032
I20260925 15:19:31.106892 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:31.107097 8366 external_mini_cluster.cc:1428] Setting key 222476c377db336b2492b9037819f458
I20260925 15:19:31.109894 28099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6: Bootstrap starting.
I20260925 15:19:31.110577 28099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:31.110863 28099 log.cc:824] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:31.111635 28099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6: No bootstrap required, opened a new log
I20260925 15:19:31.112732 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.107657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.8.43.190:38215)
0925 15:19:31.107848 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.107855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.107880 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:19:31.108314 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.108321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.108384 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.108501 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:31.108730 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.109592 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.110694 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.111165 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.111312 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.111995 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:31.112000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:31.112000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:31.112204 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:31.112245 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:31.112259 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:31.112327 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.112417 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.112513 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260925 15:19:31.114059 28099 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41bddeff2234910a728902aca6359e6" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38215 } }
I20260925 15:19:31.114189 28099 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:31.114212 28099 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41bddeff2234910a728902aca6359e6, State: Initialized, Role: FOLLOWER
I20260925 15:19:31.114331 28099 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41bddeff2234910a728902aca6359e6" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38215 } }
I20260925 15:19:31.114409 28099 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:31.114432 28099 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:31.114485 28099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:31.115065 28099 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41bddeff2234910a728902aca6359e6" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38215 } }
I20260925 15:19:31.115150 28099 leader_election.cc:302] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41bddeff2234910a728902aca6359e6; no voters:
I20260925 15:19:31.115360 28099 leader_election.cc:288] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:31.115439 28104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:31.115607 28104 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [term 1 LEADER]: Becoming Leader. State: Replica: d41bddeff2234910a728902aca6359e6, State: Running, Role: LEADER
I20260925 15:19:31.115636 28099 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:31.115707 28104 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41bddeff2234910a728902aca6359e6" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38215 } }
I20260925 15:19:31.116257 28106 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41bddeff2234910a728902aca6359e6. Latest consensus state: current_term: 1 leader_uuid: "d41bddeff2234910a728902aca6359e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41bddeff2234910a728902aca6359e6" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38215 } } }
I20260925 15:19:31.116365 28106 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:31.116253 28105 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41bddeff2234910a728902aca6359e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41bddeff2234910a728902aca6359e6" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38215 } } }
I20260925 15:19:31.116732 28113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:31.116780 28105 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:31.117285 28113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:31.118753 28113 catalog_manager.cc:1347] Generated new cluster ID: 1e3ed51429814fcab3cd561ccf8fdd61
I20260925 15:19:31.118811 28113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:31.124799 28113 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:31.125488 28113 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:31.130200 28113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6: Generated new TSK 0
I20260925 15:19:31.130352 28113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:31.135520 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.129:0
--local_ip_for_outbound_sockets=127.8.43.129
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38215
--builtin_ntp_servers=127.8.43.148:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:19:31.217308 28123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:31.217494 28123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:31.217515 28123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:31.218691 28123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:31.218731 28123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:31.218756 28123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.129
I20260925 15:19:31.220112 28123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.43.129
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.129
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:31.220350 28123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:31.220641 28123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:31.225544 28129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.225651 28131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.225556 28128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:31.225678 28123 server_base.cc:1034] running on GCE node
I20260925 15:19:31.226054 28123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:31.226326 28123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:31.227487 28123 hybrid_clock.cc:648] HybridClock initialized: now 1790349571227467 us; error 38 us; skew 500 ppm
I20260925 15:19:31.228783 28123 webserver.cc:460] Webserver started at http://127.8.43.129:46559/ using document root <none> and password file <none>
I20260925 15:19:31.229040 28123 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:31.229090 28123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:31.229264 28123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:31.230345 28123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/data/instance:
uuid: "56a720bd058b49ec86dc18671f1864c3"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "bf20cb27873c80f9cd122fb92c801b2d"
server_key_iv: "adef317abdcf16505226b09a35aef43e"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.230708 28123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance:
uuid: "56a720bd058b49ec86dc18671f1864c3"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "bf20cb27873c80f9cd122fb92c801b2d"
server_key_iv: "adef317abdcf16505226b09a35aef43e"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.232188 28123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:31.233009 28137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.233255 28123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:31.233366 28123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/wal
uuid: "56a720bd058b49ec86dc18671f1864c3"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "bf20cb27873c80f9cd122fb92c801b2d"
server_key_iv: "adef317abdcf16505226b09a35aef43e"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.233456 28123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:31.250905 28123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:31.251293 28123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:31.251462 28123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:31.251706 28123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:31.252049 28123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:31.252103 28123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.252136 28123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:31.252163 28123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.258486 28123 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.129:43183
I20260925 15:19:31.258530 28250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.129:43183 every 8 connection(s)
I20260925 15:19:31.258898 28123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/data/info.pb
I20260925 15:19:31.262097 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 28123
I20260925 15:19:31.262217 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-0/wal/instance
I20260925 15:19:31.262400 8366 external_mini_cluster.cc:1428] Setting key 950ae10dad16aad3e738059306aa3107
I20260925 15:19:31.263396 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.130:0
--local_ip_for_outbound_sockets=127.8.43.130
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38215
--builtin_ntp_servers=127.8.43.148:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:19:31.264426 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.259681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.129:38783 (local address 127.8.43.190:38215)
0925 15:19:31.259723 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.259726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.259952 (+ 226us) server_negotiation.cc:407] Connection header received
0925 15:19:31.260510 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.260511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.260528 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.260583 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:31.261273 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.261791 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.262914 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.263167 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.263278 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.264159 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:31.264160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:31.264161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:31.264165 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:31.264178 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:31.264182 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:31.264226 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.264318 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.264347 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:19:31.264674 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.259647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38215 (local address 127.8.43.129:38783)
0925 15:19:31.259828 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.259863 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.260385 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.260652 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.260656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.260672 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:31.261173 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.261178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.261891 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.261892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.262814 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.262817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.263268 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.263269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.263411 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.264013 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:31.264013 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:31.264042 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:31.264064 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:31.264225 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:31.264230 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:31.264233 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.264322 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.264423 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260925 15:19:31.265139 28251 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38215
I20260925 15:19:31.265244 28251 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:31.265440 28251 heartbeater.cc:511] Master 127.8.43.190:38215 requested a full tablet report, sending...
I20260925 15:19:31.265937 28063 ts_manager.cc:194] Registered new tserver with Master: 56a720bd058b49ec86dc18671f1864c3 (127.8.43.129:43183)
I20260925 15:19:31.266639 28063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.129:38783
W20260925 15:19:31.344452 28254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:31.344696 28254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:31.344722 28254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:31.345831 28254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:31.345870 28254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:31.345893 28254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.130
I20260925 15:19:31.347126 28254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.43.130
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.130
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:31.347357 28254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:31.347596 28254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:31.352312 28262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.352388 28260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.352348 28259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:31.352638 28254 server_base.cc:1034] running on GCE node
I20260925 15:19:31.352869 28254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:31.353107 28254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:31.354262 28254 hybrid_clock.cc:648] HybridClock initialized: now 1790349571354261 us; error 39 us; skew 500 ppm
I20260925 15:19:31.355580 28254 webserver.cc:460] Webserver started at http://127.8.43.130:40257/ using document root <none> and password file <none>
I20260925 15:19:31.355818 28254 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:31.355887 28254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:31.356035 28254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:31.357077 28254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/data/instance:
uuid: "5cc61db3590642ea993c011dc5b863ce"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "94c544bc1c4ca7a7e30fe29a51fbdcb6"
server_key_iv: "7915033d3be3cc1e7ab571bb6509dc5c"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.357425 28254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance:
uuid: "5cc61db3590642ea993c011dc5b863ce"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "94c544bc1c4ca7a7e30fe29a51fbdcb6"
server_key_iv: "7915033d3be3cc1e7ab571bb6509dc5c"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.358772 28254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:31.359542 28268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.359704 28254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:31.359833 28254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/wal
uuid: "5cc61db3590642ea993c011dc5b863ce"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "94c544bc1c4ca7a7e30fe29a51fbdcb6"
server_key_iv: "7915033d3be3cc1e7ab571bb6509dc5c"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.359907 28254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:31.369243 28254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:31.369521 28254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:31.369666 28254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:31.369879 28254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:31.370199 28254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:31.370251 28254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.370291 28254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:31.370318 28254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.376156 28254 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.130:46813
I20260925 15:19:31.376210 28381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.130:46813 every 8 connection(s)
I20260925 15:19:31.376570 28254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/data/info.pb
I20260925 15:19:31.378539 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 28254
I20260925 15:19:31.378641 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/wal/instance
I20260925 15:19:31.378887 8366 external_mini_cluster.cc:1428] Setting key beef6e9636668d8dc925c8b07bd1f69c
I20260925 15:19:31.379921 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.43.131:0
--local_ip_for_outbound_sockets=127.8.43.131
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38215
--builtin_ntp_servers=127.8.43.148:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:19:31.382920 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.377365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.130:49289 (local address 127.8.43.190:38215)
0925 15:19:31.377412 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.377414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.377641 (+ 227us) server_negotiation.cc:407] Connection header received
0925 15:19:31.378187 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.378188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.378209 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.378297 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:31.379272 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.379837 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.381380 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.381698 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.381777 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.382601 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:31.382603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:31.382603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:31.382608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:31.382624 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:31.382628 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:31.382686 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.382775 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.382801 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:19:31.383021 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.377300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38215 (local address 127.8.43.130:49289)
0925 15:19:31.377495 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.377538 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.378023 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.378410 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.378415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.378426 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:31.379160 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.379165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.379923 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.379924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.381247 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.381250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.381784 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.381785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.381892 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.382455 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:31.382456 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:31.382490 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:31.382513 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:31.382695 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:31.382700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:31.382702 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.382753 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.382831 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260925 15:19:31.383509 28382 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38215
I20260925 15:19:31.383621 28382 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:31.383795 28382 heartbeater.cc:511] Master 127.8.43.190:38215 requested a full tablet report, sending...
I20260925 15:19:31.384164 28063 ts_manager.cc:194] Registered new tserver with Master: 5cc61db3590642ea993c011dc5b863ce (127.8.43.130:46813)
I20260925 15:19:31.384544 28063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.130:49289
W20260925 15:19:31.461067 28385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:31.461303 28385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:31.461340 28385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:31.462576 28385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:31.462630 28385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:31.462672 28385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.43.131
I20260925 15:19:31.464135 28385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.43.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.43.131
--webserver_port=0
--tserver_master_addrs=127.8.43.190:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.43.131
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:31.464382 28385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:31.464665 28385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:31.469515 28393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.469556 28390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:31.469515 28391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:31.469727 28385 server_base.cc:1034] running on GCE node
I20260925 15:19:31.469928 28385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:31.470116 28385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:31.471284 28385 hybrid_clock.cc:648] HybridClock initialized: now 1790349571471261 us; error 43 us; skew 500 ppm
I20260925 15:19:31.472536 28385 webserver.cc:460] Webserver started at http://127.8.43.131:39651/ using document root <none> and password file <none>
I20260925 15:19:31.472780 28385 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:31.472847 28385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:31.472996 28385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:31.474077 28385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/data/instance:
uuid: "77a49eb06dc64232a36d6718ebc0d81c"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "f03bfda3af6d321d23088b576073bfe7"
server_key_iv: "746828dd0c8664eb50549e03bab1ae61"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.474426 28385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance:
uuid: "77a49eb06dc64232a36d6718ebc0d81c"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "f03bfda3af6d321d23088b576073bfe7"
server_key_iv: "746828dd0c8664eb50549e03bab1ae61"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.475876 28385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:31.476581 28399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.476748 28385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:31.476848 28385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/wal
uuid: "77a49eb06dc64232a36d6718ebc0d81c"
format_stamp: "Formatted at 2026-09-25 15:19:31 on dist-test-slave-c5d6"
server_key: "f03bfda3af6d321d23088b576073bfe7"
server_key_iv: "746828dd0c8664eb50549e03bab1ae61"
server_key_version: "encryptionkey@0"
I20260925 15:19:31.476920 28385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:31.499670 28385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:31.499994 28385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:31.500150 28385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:31.500452 28385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:19:31.500811 28385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:19:31.500864 28385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.500900 28385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:19:31.500927 28385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:31.506597 28385 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.131:40601
I20260925 15:19:31.506644 28512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.131:40601 every 8 connection(s)
I20260925 15:19:31.506953 28385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/data/info.pb
I20260925 15:19:31.511869 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.507641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:50849 (local address 127.8.43.190:38215)
0925 15:19:31.507708 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.507710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.507954 (+ 244us) server_negotiation.cc:407] Connection header received
0925 15:19:31.508427 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.508428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.508437 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.508508 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:31.509174 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.509659 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.510643 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.510827 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.510881 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.511646 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:31.511648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:31.511648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:31.511652 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:31.511666 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:31.511670 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:31.511713 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.511791 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.511812 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:19:31.512161 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.507633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38215 (local address 127.8.43.131:50849)
0925 15:19:31.507826 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.507866 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.508349 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.508567 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.508570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.508579 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:31.509098 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.509103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.509743 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.509744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.510565 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.510567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.510904 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.510905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.511027 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.511521 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:19:31.511521 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:31.511549 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:19:31.511570 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:31.511725 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:31.511729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:31.511731 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.511782 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.511858 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260925 15:19:31.512645 28513 heartbeater.cc:347] Connected to a master server at 127.8.43.190:38215
I20260925 15:19:31.512748 28513 heartbeater.cc:464] Registering TS with master...
I20260925 15:19:31.512923 28513 heartbeater.cc:511] Master 127.8.43.190:38215 requested a full tablet report, sending...
I20260925 15:19:31.513294 28063 ts_manager.cc:194] Registered new tserver with Master: 77a49eb06dc64232a36d6718ebc0d81c (127.8.43.131:40601)
I20260925 15:19:31.513652 28063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.43.131:50849
I20260925 15:19:31.515393 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 28385
I20260925 15:19:31.515506 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/wal/instance
I20260925 15:19:31.515735 8366 external_mini_cluster.cc:1428] Setting key da11d789854718370922a17d4a5995cd
I20260925 15:19:31.516642 8366 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:19:31.520810 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.518107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.8.43.190:38215)
0925 15:19:31.518146 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.518148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.518263 (+ 115us) server_negotiation.cc:407] Connection header received
0925 15:19:31.518304 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.518304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.518312 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.518362 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:31.518497 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.518963 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.519749 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.519934 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.520002 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.520628 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:31.520629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:31.520629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:31.520633 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:31.520645 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:31.520649 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:31.520690 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.520741 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.520758 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:19:31.522533 28063 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:19:31.524260 28063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:19:31.530424 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.526436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:43183 (local address 127.0.0.1:36486)
0925 15:19:31.526720 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.526770 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.526840 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.527177 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.527181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.527184 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:31.527273 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.527276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.528209 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.528210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.529339 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.529342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.530139 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.530140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.530217 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.530231 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.530298 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.530348 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
I20260925 15:19:31.530467 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.526638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.8.43.129:43183)
0925 15:19:31.526851 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.526857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.526869 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:19:31.526915 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.526918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.526935 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.527050 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:31.527359 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.528124 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.529420 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.530054 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.530128 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.530217 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.530324 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.530364 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
I20260925 15:19:31.530820 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.526441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.131:40601 (local address 127.0.0.1:49342)
0925 15:19:31.527037 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.527047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.527107 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.527390 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.527391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.527393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:31.527436 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.527438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.528667 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.528668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.529614 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.529616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.530562 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.530563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.530625 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.530628 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.530710 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.530750 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":112,"threads_started":1}
I20260925 15:19:31.530844 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.526862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.8.43.131:40601)
0925 15:19:31.527003 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.527007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.527165 (+ 158us) server_negotiation.cc:407] Connection header received
0925 15:19:31.527206 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.527208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.527245 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.527322 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:31.527509 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.528575 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.529687 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.530384 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.530446 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.530524 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.530725 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.530761 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260925 15:19:31.531592 28185 tablet_service.cc:1467] Processing CreateTablet for tablet 916388532b684a3da23b5614699c2861 (DEFAULT_TABLE table=test-table [id=1f834f958166423cb01e44eb98371aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:31.531590 28447 tablet_service.cc:1467] Processing CreateTablet for tablet 916388532b684a3da23b5614699c2861 (DEFAULT_TABLE table=test-table [id=1f834f958166423cb01e44eb98371aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:31.531766 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.526436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:46813 (local address 127.0.0.1:59578)
0925 15:19:31.526942 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.526951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.527039 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.527303 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.527304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.527307 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:31.527352 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.527354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.529681 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.529682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.530461 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.530464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.531549 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.531550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.531593 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.531596 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.531661 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.531699 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":109,"threads_started":1}
I20260925 15:19:31.531952 28525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.526717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.8.43.130:46813)
0925 15:19:31.526898 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.526903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.527012 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:19:31.527127 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.527130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.527144 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.527197 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:31.527442 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.528589 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.530565 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.531361 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.531409 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.531491 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.531679 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.531716 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260925 15:19:31.531956 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916388532b684a3da23b5614699c2861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:31.531956 28447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916388532b684a3da23b5614699c2861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:31.535396 28527 tablet_bootstrap.cc:492] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Bootstrap starting.
I20260925 15:19:31.535396 28528 tablet_bootstrap.cc:492] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3: Bootstrap starting.
I20260925 15:19:31.536142 28527 tablet_bootstrap.cc:654] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:31.536165 28316 tablet_service.cc:1467] Processing CreateTablet for tablet 916388532b684a3da23b5614699c2861 (DEFAULT_TABLE table=test-table [id=1f834f958166423cb01e44eb98371aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:19:31.536471 28527 log.cc:824] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:31.536541 28316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916388532b684a3da23b5614699c2861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:31.537129 28528 tablet_bootstrap.cc:654] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:31.537248 28527 tablet_bootstrap.cc:492] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: No bootstrap required, opened a new log
I20260925 15:19:31.537317 28527 ts_tablet_manager.cc:1397] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:31.537442 28528 log.cc:824] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:31.538211 28528 tablet_bootstrap.cc:492] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3: No bootstrap required, opened a new log
I20260925 15:19:31.538281 28528 ts_tablet_manager.cc:1397] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 15:19:31.539659 28527 raft_consensus.cc:348] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.539825 28527 raft_consensus.cc:374] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:31.539866 28527 raft_consensus.cc:729] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a49eb06dc64232a36d6718ebc0d81c, State: Initialized, Role: FOLLOWER
I20260925 15:19:31.539999 28527 consensus_queue.cc:260] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.540258 28531 tablet_bootstrap.cc:492] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Bootstrap starting.
I20260925 15:19:31.540288 28527 ts_tablet_manager.cc:1428] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:19:31.540400 28513 heartbeater.cc:503] Master 127.8.43.190:38215 was elected leader, sending a full tablet report...
I20260925 15:19:31.540915 28531 tablet_bootstrap.cc:654] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:31.541214 28531 log.cc:824] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:31.541237 28528 raft_consensus.cc:348] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.541386 28528 raft_consensus.cc:374] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:31.541425 28528 raft_consensus.cc:729] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a720bd058b49ec86dc18671f1864c3, State: Initialized, Role: FOLLOWER
I20260925 15:19:31.541539 28528 consensus_queue.cc:260] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.541831 28528 ts_tablet_manager.cc:1428] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260925 15:19:31.541867 28251 heartbeater.cc:503] Master 127.8.43.190:38215 was elected leader, sending a full tablet report...
I20260925 15:19:31.542066 28531 tablet_bootstrap.cc:492] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: No bootstrap required, opened a new log
I20260925 15:19:31.542152 28531 ts_tablet_manager.cc:1397] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:19:31.544023 28531 raft_consensus.cc:348] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.544160 28531 raft_consensus.cc:374] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:31.544205 28531 raft_consensus.cc:729] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc61db3590642ea993c011dc5b863ce, State: Initialized, Role: FOLLOWER
I20260925 15:19:31.544338 28531 consensus_queue.cc:260] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.544584 28531 ts_tablet_manager.cc:1428] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:19:31.544616 28382 heartbeater.cc:503] Master 127.8.43.190:38215 was elected leader, sending a full tablet report...
W20260925 15:19:31.627107 28383 tablet.cc:2369] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:31.728170 28532 raft_consensus.cc:482] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:19:31.728292 28532 raft_consensus.cc:504] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.728719 28532 leader_election.cc:288] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56a720bd058b49ec86dc18671f1864c3 (127.8.43.129:43183), 5cc61db3590642ea993c011dc5b863ce (127.8.43.130:46813)
I20260925 15:19:31.732034 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.728979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.130:46813 (local address 127.8.43.131:40471)
0925 15:19:31.729039 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.729058 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.729109 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.729366 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.729367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.729369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:31.729448 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.729450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.730273 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.730274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.731085 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.731087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.731797 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.731797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.731871 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.731873 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.731915 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.731945 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":33}
I20260925 15:19:31.732076 28525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.729110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:40471 (local address 127.8.43.130:46813)
0925 15:19:31.729169 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.729173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.729183 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:19:31.729222 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.729223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.729234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.729321 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:31.729548 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.730153 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.731190 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.731705 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.731798 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.731809 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.731949 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.731999 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:19:31.732144 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.729065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.43.131:45705 (local address 127.8.43.129:43183)
0925 15:19:31.729118 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.729120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.729314 (+ 194us) server_negotiation.cc:407] Connection header received
0925 15:19:31.729393 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.729394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.729406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.729468 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:19:31.729679 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.730127 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.731366 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.731713 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.731802 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.731809 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.732026 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.732074 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:19:31.732163 28536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.728982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:43183 (local address 127.8.43.131:45705)
0925 15:19:31.729230 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.729242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:31.729307 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:31.729489 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:31.729490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:31.729492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:19:31.729594 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.729597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.730219 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.730220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.731186 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:31.731189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.731824 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.731825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:31.731941 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.731947 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:19:31.732014 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:19:31.732054 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":73,"thread_start_us":95,"threads_started":1}
I20260925 15:19:31.732399 28336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916388532b684a3da23b5614699c2861" candidate_uuid: "77a49eb06dc64232a36d6718ebc0d81c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc61db3590642ea993c011dc5b863ce" is_pre_election: true
I20260925 15:19:31.732496 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916388532b684a3da23b5614699c2861" candidate_uuid: "77a49eb06dc64232a36d6718ebc0d81c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a720bd058b49ec86dc18671f1864c3" is_pre_election: true
I20260925 15:19:31.732615 28336 raft_consensus.cc:2454] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a49eb06dc64232a36d6718ebc0d81c in term 0.
I20260925 15:19:31.732700 28205 raft_consensus.cc:2454] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a49eb06dc64232a36d6718ebc0d81c in term 0.
I20260925 15:19:31.732904 28403 leader_election.cc:302] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc61db3590642ea993c011dc5b863ce, 77a49eb06dc64232a36d6718ebc0d81c; no voters:
I20260925 15:19:31.733021 28532 raft_consensus.cc:2781] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:19:31.733069 28532 raft_consensus.cc:482] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:19:31.733091 28532 raft_consensus.cc:3037] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:31.733758 28532 raft_consensus.cc:504] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.733887 28532 leader_election.cc:288] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [CANDIDATE]: Term 1 election: Requested vote from peers 56a720bd058b49ec86dc18671f1864c3 (127.8.43.129:43183), 5cc61db3590642ea993c011dc5b863ce (127.8.43.130:46813)
I20260925 15:19:31.734040 28336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916388532b684a3da23b5614699c2861" candidate_uuid: "77a49eb06dc64232a36d6718ebc0d81c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc61db3590642ea993c011dc5b863ce"
I20260925 15:19:31.734104 28205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916388532b684a3da23b5614699c2861" candidate_uuid: "77a49eb06dc64232a36d6718ebc0d81c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a720bd058b49ec86dc18671f1864c3"
I20260925 15:19:31.734145 28336 raft_consensus.cc:3037] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:31.734169 28205 raft_consensus.cc:3037] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:31.735021 28336 raft_consensus.cc:2454] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a49eb06dc64232a36d6718ebc0d81c in term 1.
I20260925 15:19:31.735021 28205 raft_consensus.cc:2454] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a49eb06dc64232a36d6718ebc0d81c in term 1.
I20260925 15:19:31.735257 28403 leader_election.cc:302] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cc61db3590642ea993c011dc5b863ce, 77a49eb06dc64232a36d6718ebc0d81c; no voters:
I20260925 15:19:31.735351 28532 raft_consensus.cc:2781] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:31.735492 28532 raft_consensus.cc:686] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 1 LEADER]: Becoming Leader. State: Replica: 77a49eb06dc64232a36d6718ebc0d81c, State: Running, Role: LEADER
I20260925 15:19:31.735594 28532 consensus_queue.cc:237] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } }
I20260925 15:19:31.736603 28063 catalog_manager.cc:5495] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77a49eb06dc64232a36d6718ebc0d81c (127.8.43.131). New cstate: current_term: 1 leader_uuid: "77a49eb06dc64232a36d6718ebc0d81c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a49eb06dc64232a36d6718ebc0d81c" member_type: VOTER last_known_addr { host: "127.8.43.131" port: 40601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 } health_report { overall_health: UNKNOWN } } }
W20260925 15:19:31.757445 28514 tablet.cc:2369] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:31.759495 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.756940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.8.43.131:40601)
0925 15:19:31.757000 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:31.757003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:31.757043 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:19:31.757089 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:31.757090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:31.757106 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:31.757181 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:31.757312 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.757859 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.758636 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:31.758846 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:31.758923 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:31.759092 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:31.759193 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:31.759342 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:31.759367 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 15:19:31.759385 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 15:19:31.759675 28252 tablet.cc:2369] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:19:31.761428 28205 raft_consensus.cc:1261] T 916388532b684a3da23b5614699c2861 P 56a720bd058b49ec86dc18671f1864c3 [term 1 FOLLOWER]: Refusing update from remote peer 77a49eb06dc64232a36d6718ebc0d81c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:31.761721 28336 raft_consensus.cc:1261] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 1 FOLLOWER]: Refusing update from remote peer 77a49eb06dc64232a36d6718ebc0d81c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:31.761812 28537 consensus_queue.cc:1035] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a720bd058b49ec86dc18671f1864c3" member_type: VOTER last_known_addr { host: "127.8.43.129" port: 43183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:31.762101 28537 consensus_queue.cc:1035] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc61db3590642ea993c011dc5b863ce" member_type: VOTER last_known_addr { host: "127.8.43.130" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:31.764267 28540 mvcc.cc:205] Tried to move back new op lower bound from 7333271845932216320 to 7333271845829849088. Current Snapshot: MvccSnapshot[applied={T|T < 7333271845932216320}]
I20260925 15:19:31.765095 28541 mvcc.cc:205] Tried to move back new op lower bound from 7333271845932216320 to 7333271845829849088. Current Snapshot: MvccSnapshot[applied={T|T < 7333271845932216320}]
I20260925 15:19:31.765213 28543 mvcc.cc:205] Tried to move back new op lower bound from 7333271845932216320 to 7333271845829849088. Current Snapshot: MvccSnapshot[applied={T|T < 7333271845932216320}]
I20260925 15:19:31.766570 28062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:31.766657 28062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:19:31.767372 28062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d41bddeff2234910a728902aca6359e6: Sending DeleteTablet for 3 replicas of tablet 916388532b684a3da23b5614699c2861
I20260925 15:19:31.767668 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet 916388532b684a3da23b5614699c2861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:31 UTC) from {username='slave'} at 127.0.0.1:36486
I20260925 15:19:31.767684 28447 tablet_service.cc:1514] Processing DeleteTablet for tablet 916388532b684a3da23b5614699c2861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:31 UTC) from {username='slave'} at 127.0.0.1:49342
I20260925 15:19:31.767875 28549 tablet_replica.cc:330] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: stopping tablet replica
I20260925 15:19:31.767894 28316 tablet_service.cc:1514] Processing DeleteTablet for tablet 916388532b684a3da23b5614699c2861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:19:31 UTC) from {username='slave'} at 127.0.0.1:59578
I20260925 15:19:31.768059 28549 raft_consensus.cc:2229] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:19:31.768087 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 28123
I20260925 15:19:31.768105 28551 tablet_replica.cc:330] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: stopping tablet replica
I20260925 15:19:31.768189 28551 raft_consensus.cc:2229] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:19:31.768296 28551 raft_consensus.cc:2258] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:31.768318 28549 raft_consensus.cc:2258] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:19:31.768935 28551 ts_tablet_manager.cc:1905] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:31.770207 28549 ts_tablet_manager.cc:1905] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:19:31.771247 28551 ts_tablet_manager.cc:1918] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:31.771327 28551 log.cc:1196] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-1/wal/wals/916388532b684a3da23b5614699c2861
I20260925 15:19:31.771600 28551 ts_tablet_manager.cc:1939] T 916388532b684a3da23b5614699c2861 P 5cc61db3590642ea993c011dc5b863ce: Deleting consensus metadata
I20260925 15:19:31.771993 28050 catalog_manager.cc:4841] TS 5cc61db3590642ea993c011dc5b863ce (127.8.43.130:46813): tablet 916388532b684a3da23b5614699c2861 (table test-table [id=1f834f958166423cb01e44eb98371aeb]) successfully deleted
I20260925 15:19:31.772153 28549 ts_tablet_manager.cc:1918] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:19:31.772226 28549 log.cc:1196] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Deleting WAL directory at /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349306597827-8366-0/minicluster-data/ts-2/wal/wals/916388532b684a3da23b5614699c2861
I20260925 15:19:31.772476 28549 ts_tablet_manager.cc:1939] T 916388532b684a3da23b5614699c2861 P 77a49eb06dc64232a36d6718ebc0d81c: Deleting consensus metadata
I20260925 15:19:31.772883 28049 catalog_manager.cc:4841] TS 77a49eb06dc64232a36d6718ebc0d81c (127.8.43.131:40601): tablet 916388532b684a3da23b5614699c2861 (table test-table [id=1f834f958166423cb01e44eb98371aeb]) successfully deleted
W20260925 15:19:31.775497 28048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.43.129:43183 (error 108)
I20260925 15:19:31.775682 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 28254
I20260925 15:19:31.776484 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:31.776316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.129:43183 (local address 127.0.0.1:36490)
0925 15:19:31.776379 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:31.776434 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.129:43183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:19:31.776600 28048 catalog_manager.cc:4584] TS 56a720bd058b49ec86dc18671f1864c3 (127.8.43.129:43183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 916388532b684a3da23b5614699c2861: Network error: Client connection negotiation failed: client connection to 127.8.43.129:43183: connect: Connection refused (error 111)
I20260925 15:19:31.781880 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 28385
I20260925 15:19:31.786530 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 28032
2026-09-25T15:19:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (830 ms)
[----------] 6 tests from EncryptionPolicyTest (8384 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:31 dist-test-slave-c5d6 krb5kdc[28555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:31 dist-test-slave-c5d6 krb5kdc[28555](info): set up 2 sockets
Sep 25 15:19:31 dist-test-slave-c5d6 krb5kdc[28555](info): commencing operation
krb5kdc: starting...
W20260925 15:19:32.819288 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:32 dist-test-slave-c5d6 krb5kdc[28555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:32.936147 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:33.016928 28568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:33.017138 28568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:33.017160 28568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:33.018260 28568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:33.018296 28568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:33.018311 28568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:33.018323 28568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:33.018335 28568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:33.019793 28568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:33.019987 28568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:33.020174 28568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:33.025071 28574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:33.025079 28573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:33.025295 28568 server_base.cc:1034] running on GCE node
W20260925 15:19:33.025079 28576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:33.025602 28568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:33.025926 28568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:33.027098 28568 hybrid_clock.cc:648] HybridClock initialized: now 1790349573027076 us; error 37 us; skew 500 ppm
Sep 25 15:19:33 dist-test-slave-c5d6 krb5kdc[28555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:33.029680 28568 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:33.030066 28568 webserver.cc:460] Webserver started at http://127.0.0.1:40351/ using document root <none> and password file <none>
I20260925 15:19:33.030325 28568 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:33.030373 28568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:33.030534 28568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:33.031674 28568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "a0a1e995b5ca48c8a924171b858e9fdd"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-c5d6"
server_key: "38a29223c5a0534e11f32575c35ee54a"
server_key_iv: "54e4f269b26d704f047178e7a8aee56e"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.032032 28568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "a0a1e995b5ca48c8a924171b858e9fdd"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-c5d6"
server_key: "38a29223c5a0534e11f32575c35ee54a"
server_key_iv: "54e4f269b26d704f047178e7a8aee56e"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.033424 28568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:33.034281 28583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:33.034520 28568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:33.034624 28568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "a0a1e995b5ca48c8a924171b858e9fdd"
format_stamp: "Formatted at 2026-09-25 15:19:33 on dist-test-slave-c5d6"
server_key: "38a29223c5a0534e11f32575c35ee54a"
server_key_iv: "54e4f269b26d704f047178e7a8aee56e"
server_key_version: "encryptionkey@0"
I20260925 15:19:33.034706 28568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:33.082103 28568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:33.084560 28568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:33.084719 28568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:33.088603 28568 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46701
I20260925 15:19:33.088652 28635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46701 every 8 connection(s)
I20260925 15:19:33.089030 28568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:33.089962 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:33.091241 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 28568
I20260925 15:19:33.091357 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:33.091535 8366 external_mini_cluster.cc:1428] Setting key 1288b809ef8a79643bd90f5fe974cf60
I20260925 15:19:33.092772 28636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd: Bootstrap starting.
I20260925 15:19:33.093446 28636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:33.093763 28636 log.cc:824] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:33.094511 28636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd: No bootstrap required, opened a new log
I20260925 15:19:33.096700 28636 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a1e995b5ca48c8a924171b858e9fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46701 } }
I20260925 15:19:33.096868 28636 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:33.096915 28636 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a1e995b5ca48c8a924171b858e9fdd, State: Initialized, Role: FOLLOWER
I20260925 15:19:33.097038 28636 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a1e995b5ca48c8a924171b858e9fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46701 } }
I20260925 15:19:33.097116 28636 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:33.097172 28636 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:33.097220 28636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:33.097720 28636 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a1e995b5ca48c8a924171b858e9fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46701 } }
I20260925 15:19:33.097800 28636 leader_election.cc:302] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0a1e995b5ca48c8a924171b858e9fdd; no voters:
I20260925 15:19:33.097975 28636 leader_election.cc:288] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:33.097998 28641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:33.098121 28641 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [term 1 LEADER]: Becoming Leader. State: Replica: a0a1e995b5ca48c8a924171b858e9fdd, State: Running, Role: LEADER
I20260925 15:19:33.098199 28641 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a1e995b5ca48c8a924171b858e9fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46701 } }
I20260925 15:19:33.098227 28636 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:33.098532 28641 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0a1e995b5ca48c8a924171b858e9fdd. Latest consensus state: current_term: 1 leader_uuid: "a0a1e995b5ca48c8a924171b858e9fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a1e995b5ca48c8a924171b858e9fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46701 } } }
I20260925 15:19:33.098606 28641 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:33.098551 28643 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0a1e995b5ca48c8a924171b858e9fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a1e995b5ca48c8a924171b858e9fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46701 } } }
Sep 25 15:19:33 dist-test-slave-c5d6 krb5kdc[28555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:19:33.098778 28643 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:33.099637 28646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:33.100368 28646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:33.101958 28646 catalog_manager.cc:1347] Generated new cluster ID: c3e6eac30f734fefb693db033b1108a9
I20260925 15:19:33.102007 28646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:33.102929 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.092117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60605 (local address 127.0.0.1:46701)
0925 15:19:33.092304 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.092315 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.092350 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:19:33.092863 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.092871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.092939 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.093113 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.093709 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.095089 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.095886 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.096588 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.096703 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.099443 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.099447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.099447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.099457 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.101254 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.101744 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.101747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.101747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.101872 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.102129 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.102130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.102130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.102348 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.102532 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.102615 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.102715 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260925 15:19:33.104082 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:33.107384 28646 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:33.108147 28646 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:33.114697 28646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0a1e995b5ca48c8a924171b858e9fdd: Generated new TSK 0
I20260925 15:19:33.114876 28646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:33.117288 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:33.105533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60855 (local address 127.0.0.1:46701)
0925 15:19:33.105572 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:33.105574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:33.107054 (+ 1480us) server_negotiation.cc:407] Connection header received
0925 15:19:33.107130 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:33.107131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:33.107142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:33.107207 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:33.111761 (+ 4554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.112278 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.112994 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:33.113200 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:33.113283 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:33.114690 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:33.114691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:33.114691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:33.114697 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:33.116175 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.116591 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.116592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.116593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.116640 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:33.116858 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:33.116858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:33.116859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:33.116968 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:33.117045 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:33.117188 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 15:19:33.117234 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:19:33.119370 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 28568
W20260925 15:19:33.123152 28553 connection.cc:535] client connection to 127.0.0.1:46701 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46701): Connection reset by peer (error 104)
2026-09-25T15:19:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:33 dist-test-slave-c5d6 krb5kdc[28667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:33 dist-test-slave-c5d6 krb5kdc[28667](info): set up 2 sockets
Sep 25 15:19:33 dist-test-slave-c5d6 krb5kdc[28667](info): commencing operation
krb5kdc: starting...
W20260925 15:19:34.143509 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:34 dist-test-slave-c5d6 krb5kdc[28667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:34.262884 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:34.343056 28680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:34.343297 28680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:34.343350 28680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:34.344508 28680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:34.344573 28680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:34.344597 28680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:34.344619 28680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:34.344646 28680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:34.346071 28680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:34.346347 28680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:34.346571 28680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:34.351295 28688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:34.351306 28686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:34.351291 28685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:34.351550 28680 server_base.cc:1034] running on GCE node
I20260925 15:19:34.351785 28680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:34.352058 28680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:34.353245 28680 hybrid_clock.cc:648] HybridClock initialized: now 1790349574353244 us; error 42 us; skew 500 ppm
Sep 25 15:19:34 dist-test-slave-c5d6 krb5kdc[28667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:34.360719 28680 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:34.361200 28680 webserver.cc:460] Webserver started at http://127.0.0.1:36485/ using document root <none> and password file <none>
I20260925 15:19:34.361454 28680 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:34.361537 28680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:34.361696 28680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:34.362855 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "b1ed3feffadf45f3b07a241444a1c05e"
format_stamp: "Formatted at 2026-09-25 15:19:34 on dist-test-slave-c5d6"
server_key: "02588314c2a459ed30b2511531188f2e"
server_key_iv: "dc61b6ca792d3b5313b09110b96c6458"
server_key_version: "encryptionkey@0"
I20260925 15:19:34.363303 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "b1ed3feffadf45f3b07a241444a1c05e"
format_stamp: "Formatted at 2026-09-25 15:19:34 on dist-test-slave-c5d6"
server_key: "02588314c2a459ed30b2511531188f2e"
server_key_iv: "dc61b6ca792d3b5313b09110b96c6458"
server_key_version: "encryptionkey@0"
I20260925 15:19:34.364696 28680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:34.365504 28695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:34.365674 28680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:34.365782 28680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "b1ed3feffadf45f3b07a241444a1c05e"
format_stamp: "Formatted at 2026-09-25 15:19:34 on dist-test-slave-c5d6"
server_key: "02588314c2a459ed30b2511531188f2e"
server_key_iv: "dc61b6ca792d3b5313b09110b96c6458"
server_key_version: "encryptionkey@0"
I20260925 15:19:34.365870 28680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:34.390494 28680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:34.392499 28680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:34.392678 28680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:34.397326 28680 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42695
I20260925 15:19:34.397382 28747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42695 every 8 connection(s)
I20260925 15:19:34.397806 28680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:34.398826 28748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:34.401124 28748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e: Bootstrap starting.
I20260925 15:19:34.401695 28748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:34.401990 28748 log.cc:824] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:34.402684 28748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e: No bootstrap required, opened a new log
I20260925 15:19:34.404295 28748 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ed3feffadf45f3b07a241444a1c05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260925 15:19:34.404430 28748 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:34.404452 28748 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ed3feffadf45f3b07a241444a1c05e, State: Initialized, Role: FOLLOWER
I20260925 15:19:34.404608 28748 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ed3feffadf45f3b07a241444a1c05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260925 15:19:34.404671 28748 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:34.404693 28748 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:34.404743 28748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:34.405323 28748 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ed3feffadf45f3b07a241444a1c05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260925 15:19:34.405434 28748 leader_election.cc:302] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ed3feffadf45f3b07a241444a1c05e; no voters:
I20260925 15:19:34.405619 28748 leader_election.cc:288] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:34.405664 28751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:34.405880 28748 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:34.405880 28751 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [term 1 LEADER]: Becoming Leader. State: Replica: b1ed3feffadf45f3b07a241444a1c05e, State: Running, Role: LEADER
I20260925 15:19:34.406090 28751 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ed3feffadf45f3b07a241444a1c05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260925 15:19:34.406675 28753 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1ed3feffadf45f3b07a241444a1c05e. Latest consensus state: current_term: 1 leader_uuid: "b1ed3feffadf45f3b07a241444a1c05e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ed3feffadf45f3b07a241444a1c05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } } }
I20260925 15:19:34.406776 28753 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:34.407191 28760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:34.407779 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 28680
I20260925 15:19:34.407398 28752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1ed3feffadf45f3b07a241444a1c05e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ed3feffadf45f3b07a241444a1c05e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } } }
I20260925 15:19:34.407727 28760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:34.407910 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:34.407904 28752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:34.408083 8366 external_mini_cluster.cc:1428] Setting key 2872a93ee88e73c71a987b3f1b32a504
I20260925 15:19:34.409749 28760 catalog_manager.cc:1347] Generated new cluster ID: 93bb79687d9b47ff8e27e7c29bf66562
I20260925 15:19:34.409811 28760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:34.414350 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.408571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38083 (local address 127.0.0.1:42695)
0925 15:19:34.408878 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.408886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.408931 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:19:34.409047 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.409054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.409154 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.409309 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:34.409571 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.410992 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.412164 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.412776 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.412880 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.413625 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:34.413628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:34.413629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:34.413851 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:34.413887 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:34.413901 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:34.413986 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.414091 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.414119 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":66,"threads_started":1}
I20260925 15:19:34.415733 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:34.417347 28760 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:34.417860 28760 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:34.426946 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:34.417277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43495 (local address 127.0.0.1:42695)
0925 15:19:34.417313 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:34.417314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:34.423443 (+ 6129us) server_negotiation.cc:407] Connection header received
0925 15:19:34.423500 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:34.423501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:34.423512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:34.423578 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:34.423734 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.424534 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.425564 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:34.425891 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:34.425961 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:34.426688 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:34.426690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:34.426690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:34.426694 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:34.426710 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:34.426715 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:34.426778 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:34.426861 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:19:34.426873 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:19:34.431890 28760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1ed3feffadf45f3b07a241444a1c05e: Generated new TSK 0
I20260925 15:19:34.432122 28760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:34.434608 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 28680
W20260925 15:19:34.439599 28665 connection.cc:535] client connection to 127.0.0.1:42695 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42695): Connection reset by peer (error 104)
2026-09-25T15:19:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:34 dist-test-slave-c5d6 krb5kdc[28779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:34 dist-test-slave-c5d6 krb5kdc[28779](info): set up 2 sockets
Sep 25 15:19:34 dist-test-slave-c5d6 krb5kdc[28779](info): commencing operation
krb5kdc: starting...
W20260925 15:19:35.467288 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:35 dist-test-slave-c5d6 krb5kdc[28779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:35.583962 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:35.664121 28792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:35.664330 28792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:35.664353 28792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:35.665547 28792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:35.665596 28792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:35.665611 28792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:35.665624 28792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:35.665681 28792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:35.667135 28792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:35.667387 28792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:35.667567 28792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:35.672245 28797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:35.672246 28800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:35.672303 28798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:35.672350 28792 server_base.cc:1034] running on GCE node
I20260925 15:19:35.672679 28792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:35.672916 28792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:35.674130 28792 hybrid_clock.cc:648] HybridClock initialized: now 1790349575674120 us; error 33 us; skew 500 ppm
Sep 25 15:19:35 dist-test-slave-c5d6 krb5kdc[28779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:35.676784 28792 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:35.677287 28792 webserver.cc:460] Webserver started at http://127.0.0.1:43035/ using document root <none> and password file <none>
I20260925 15:19:35.677517 28792 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:35.677587 28792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:35.677739 28792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:35.678815 28792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "b3a21fb34408475faf45f6b8f94d0d9c"
format_stamp: "Formatted at 2026-09-25 15:19:35 on dist-test-slave-c5d6"
server_key: "2bf4f978bb0c8ac44147f866d479e4ef"
server_key_iv: "f45a70e961a6b7dab04cc6d6ac6d6cb7"
server_key_version: "encryptionkey@0"
I20260925 15:19:35.679183 28792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "b3a21fb34408475faf45f6b8f94d0d9c"
format_stamp: "Formatted at 2026-09-25 15:19:35 on dist-test-slave-c5d6"
server_key: "2bf4f978bb0c8ac44147f866d479e4ef"
server_key_iv: "f45a70e961a6b7dab04cc6d6ac6d6cb7"
server_key_version: "encryptionkey@0"
I20260925 15:19:35.680538 28792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:35.681399 28807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:35.681607 28792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:35.681711 28792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "b3a21fb34408475faf45f6b8f94d0d9c"
format_stamp: "Formatted at 2026-09-25 15:19:35 on dist-test-slave-c5d6"
server_key: "2bf4f978bb0c8ac44147f866d479e4ef"
server_key_iv: "f45a70e961a6b7dab04cc6d6ac6d6cb7"
server_key_version: "encryptionkey@0"
I20260925 15:19:35.681795 28792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:35.685168 28792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:35.687053 28792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:35.687211 28792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:35.691184 28792 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36467
I20260925 15:19:35.691210 28859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36467 every 8 connection(s)
I20260925 15:19:35.691593 28792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:35.692551 28860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:35.694790 28860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c: Bootstrap starting.
I20260925 15:19:35.695444 28860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:35.695693 28860 log.cc:824] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:35.696355 28860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c: No bootstrap required, opened a new log
I20260925 15:19:35.697966 28860 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a21fb34408475faf45f6b8f94d0d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36467 } }
I20260925 15:19:35.698125 28860 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:35.698150 28860 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a21fb34408475faf45f6b8f94d0d9c, State: Initialized, Role: FOLLOWER
I20260925 15:19:35.698290 28860 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a21fb34408475faf45f6b8f94d0d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36467 } }
I20260925 15:19:35.698375 28860 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:35.698421 28860 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:35.698470 28860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:35.698618 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 28792
I20260925 15:19:35.698753 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:35.698990 8366 external_mini_cluster.cc:1428] Setting key 01ded3529126a0ee6b6dd24cfe53cec5
I20260925 15:19:35.699280 28860 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a21fb34408475faf45f6b8f94d0d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36467 } }
I20260925 15:19:35.699391 28860 leader_election.cc:302] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3a21fb34408475faf45f6b8f94d0d9c; no voters:
I20260925 15:19:35.699579 28860 leader_election.cc:288] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:35.699605 28863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:35.699914 28860 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:35.700070 28863 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [term 1 LEADER]: Becoming Leader. State: Replica: b3a21fb34408475faf45f6b8f94d0d9c, State: Running, Role: LEADER
I20260925 15:19:35.700258 28863 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a21fb34408475faf45f6b8f94d0d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36467 } }
I20260925 15:19:35.701254 28867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3a21fb34408475faf45f6b8f94d0d9c. Latest consensus state: current_term: 1 leader_uuid: "b3a21fb34408475faf45f6b8f94d0d9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a21fb34408475faf45f6b8f94d0d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36467 } } }
I20260925 15:19:35.701375 28867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:35.701254 28865 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3a21fb34408475faf45f6b8f94d0d9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a21fb34408475faf45f6b8f94d0d9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36467 } } }
I20260925 15:19:35.701532 28865 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:35.702065 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.699751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40879 (local address 127.0.0.1:36467)
0925 15:19:35.699892 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.699901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.699933 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:19:35.700041 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.700048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.700080 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.700501 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.701405 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.701409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.701410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:35.701415 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.701465 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:35.701481 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.701778 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.701811 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.701853 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
W20260925 15:19:35.702489 28881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:35.702545 28881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:35.702656 28875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:35.702831 28875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:35.703006 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:35.704917 28875 catalog_manager.cc:1347] Generated new cluster ID: 4abe884e3990405daeeae021fd0da55b
I20260925 15:19:35.704977 28875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:35.708352 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:35.704560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44349 (local address 127.0.0.1:36467)
0925 15:19:35.704629 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:35.704631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:35.707403 (+ 2772us) server_negotiation.cc:407] Connection header received
0925 15:19:35.707479 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:35.707481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:35.707496 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:35.707545 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:35.708129 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:35.708130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:35.708130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:35.708134 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:35.708148 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:35.708152 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:35.708207 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:35.708277 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:19:35.708289 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:19:35.710168 28875 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:35.710289 28875 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:35.721064 28875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3a21fb34408475faf45f6b8f94d0d9c: Generated new TSK 0
I20260925 15:19:35.721248 28875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:35.726712 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 28792
2026-09-25T15:19:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:35 dist-test-slave-c5d6 krb5kdc[28891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:35 dist-test-slave-c5d6 krb5kdc[28891](info): set up 2 sockets
Sep 25 15:19:35 dist-test-slave-c5d6 krb5kdc[28891](info): commencing operation
krb5kdc: starting...
W20260925 15:19:36.779196 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:36 dist-test-slave-c5d6 krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:36.891748 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:36.970444 28904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:36.970662 28904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:36.970701 28904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:36.971889 28904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:36.971930 28904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:36.971943 28904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:36.971984 28904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:36.972016 28904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:36.973467 28904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:36.973726 28904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:36.973953 28904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:36.978909 28912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:36.978973 28909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:36.979007 28910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:36.978942 28904 server_base.cc:1034] running on GCE node
I20260925 15:19:36.979350 28904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:36.979691 28904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:36.980855 28904 hybrid_clock.cc:648] HybridClock initialized: now 1790349576980832 us; error 39 us; skew 500 ppm
Sep 25 15:19:36 dist-test-slave-c5d6 krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:36.983335 28904 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:36.983767 28904 webserver.cc:460] Webserver started at http://127.0.0.1:36127/ using document root <none> and password file <none>
I20260925 15:19:36.983964 28904 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:36.984005 28904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:36.984097 28904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:36.985070 28904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "bf4977d755ef453382b92fa3e5bb7940"
format_stamp: "Formatted at 2026-09-25 15:19:36 on dist-test-slave-c5d6"
server_key: "f7b1e2709810f539f8b85122ab0c527f"
server_key_iv: "b0080d96f242938016e3054ef7857800"
server_key_version: "encryptionkey@0"
I20260925 15:19:36.985401 28904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "bf4977d755ef453382b92fa3e5bb7940"
format_stamp: "Formatted at 2026-09-25 15:19:36 on dist-test-slave-c5d6"
server_key: "f7b1e2709810f539f8b85122ab0c527f"
server_key_iv: "b0080d96f242938016e3054ef7857800"
server_key_version: "encryptionkey@0"
I20260925 15:19:36.986630 28904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:36.987375 28919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:36.987558 28904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:36.987658 28904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "bf4977d755ef453382b92fa3e5bb7940"
format_stamp: "Formatted at 2026-09-25 15:19:36 on dist-test-slave-c5d6"
server_key: "f7b1e2709810f539f8b85122ab0c527f"
server_key_iv: "b0080d96f242938016e3054ef7857800"
server_key_version: "encryptionkey@0"
I20260925 15:19:36.987748 28904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:36.997046 28904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:36.998876 28904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:36.999032 28904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:37.002938 28904 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33177
I20260925 15:19:37.002969 28971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33177 every 8 connection(s)
I20260925 15:19:37.003386 28904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:37.004321 28972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:37.006282 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 28904
I20260925 15:19:37.006410 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:37.006600 8366 external_mini_cluster.cc:1428] Setting key dd9bc85ab23adf13d2927b0881267855
I20260925 15:19:37.006688 28972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940: Bootstrap starting.
I20260925 15:19:37.007455 28972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:37.007776 28972 log.cc:824] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:37.008514 28972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940: No bootstrap required, opened a new log
I20260925 15:19:37.010775 28972 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4977d755ef453382b92fa3e5bb7940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33177 } }
I20260925 15:19:37.010906 28972 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:37.010928 28972 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4977d755ef453382b92fa3e5bb7940, State: Initialized, Role: FOLLOWER
I20260925 15:19:37.011063 28972 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4977d755ef453382b92fa3e5bb7940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33177 } }
I20260925 15:19:37.011155 28972 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:37.011188 28972 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:37.011260 28972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:37.011926 28972 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4977d755ef453382b92fa3e5bb7940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33177 } }
I20260925 15:19:37.012036 28972 leader_election.cc:302] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf4977d755ef453382b92fa3e5bb7940; no voters:
I20260925 15:19:37.012192 28972 leader_election.cc:288] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:37.012243 28977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:37.012501 28972 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:37.013425 28977 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [term 1 LEADER]: Becoming Leader. State: Replica: bf4977d755ef453382b92fa3e5bb7940, State: Running, Role: LEADER
I20260925 15:19:37.013571 28977 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4977d755ef453382b92fa3e5bb7940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33177 } }
Sep 25 15:19:37 dist-test-slave-c5d6 krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:19:37.014093 28979 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf4977d755ef453382b92fa3e5bb7940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4977d755ef453382b92fa3e5bb7940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33177 } } }
I20260925 15:19:37.014169 28979 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:37.014314 28978 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4977d755ef453382b92fa3e5bb7940. Latest consensus state: current_term: 1 leader_uuid: "bf4977d755ef453382b92fa3e5bb7940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4977d755ef453382b92fa3e5bb7940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33177 } } }
I20260925 15:19:37.014389 28978 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:37.014503 28992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:37.015049 28992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:37.016582 28992 catalog_manager.cc:1347] Generated new cluster ID: be99ed81d0494d9d8c3e6bb90c9bea86
I20260925 15:19:37.016633 28992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:37.018836 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.007183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60699 (local address 127.0.0.1:33177)
0925 15:19:37.007381 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:37.007389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:37.007424 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:19:37.008040 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:37.008056 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:37.008107 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:37.008264 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:37.008895 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.010310 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.011156 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.011798 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.011924 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:37.014756 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:37.014760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:37.014761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:37.014774 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:37.016832 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:37.017233 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:37.017236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:37.017236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:37.017339 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:37.017561 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:37.017562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:37.017563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:37.017862 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:37.018001 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:37.018445 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 15:19:37.018594 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260925 15:19:37.019930 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:37.028784 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:37.021421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47201 (local address 127.0.0.1:33177)
0925 15:19:37.021457 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:37.021467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:37.021600 (+ 133us) server_negotiation.cc:407] Connection header received
0925 15:19:37.021693 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:37.021693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:37.021703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:37.021731 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:37.022517 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.023325 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.024306 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:37.024652 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:37.024726 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:37.026021 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:37.026023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:37.026023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:37.026030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:37.027660 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:37.028033 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:37.028035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:37.028035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:37.028095 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:37.028283 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:37.028284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:37.028284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:37.028414 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:37.028517 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:37.028612 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 15:19:37.028688 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:19:37.034602 28992 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:37.035161 28992 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:37.038417 28992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf4977d755ef453382b92fa3e5bb7940: Generated new TSK 0
I20260925 15:19:37.038604 28992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:37.041211 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 28904
2026-09-25T15:19:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:37 dist-test-slave-c5d6 krb5kdc[29004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:37 dist-test-slave-c5d6 krb5kdc[29004](info): set up 2 sockets
Sep 25 15:19:37 dist-test-slave-c5d6 krb5kdc[29004](info): commencing operation
krb5kdc: starting...
W20260925 15:19:38.087457 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:38 dist-test-slave-c5d6 krb5kdc[29004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:38.205386 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:38.285147 29017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:38.285362 29017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:38.285385 29017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:38.286566 29017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:38.286609 29017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:38.286624 29017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:38.286638 29017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:38.286649 29017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:38.288198 29017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:38.288446 29017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:38.288682 29017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:38.293699 29023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:38.293699 29025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:38.293725 29022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:38.293882 29017 server_base.cc:1034] running on GCE node
I20260925 15:19:38.294214 29017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:38.294557 29017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:38.295717 29017 hybrid_clock.cc:648] HybridClock initialized: now 1790349578295710 us; error 35 us; skew 500 ppm
Sep 25 15:19:38 dist-test-slave-c5d6 krb5kdc[29004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:38.298194 29017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:38.298623 29017 webserver.cc:460] Webserver started at http://127.0.0.1:36515/ using document root <none> and password file <none>
I20260925 15:19:38.298882 29017 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:38.298950 29017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:38.299100 29017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:38.300202 29017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "d64837dba74145ff80fc136b52df64c8"
format_stamp: "Formatted at 2026-09-25 15:19:38 on dist-test-slave-c5d6"
server_key: "10e2b808a432967894226df62c305500"
server_key_iv: "d63ce3b6dbb062221b2b1480ca880d12"
server_key_version: "encryptionkey@0"
I20260925 15:19:38.300592 29017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "d64837dba74145ff80fc136b52df64c8"
format_stamp: "Formatted at 2026-09-25 15:19:38 on dist-test-slave-c5d6"
server_key: "10e2b808a432967894226df62c305500"
server_key_iv: "d63ce3b6dbb062221b2b1480ca880d12"
server_key_version: "encryptionkey@0"
I20260925 15:19:38.301885 29017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:19:38.302593 29032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:38.302757 29017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:38.302856 29017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "d64837dba74145ff80fc136b52df64c8"
format_stamp: "Formatted at 2026-09-25 15:19:38 on dist-test-slave-c5d6"
server_key: "10e2b808a432967894226df62c305500"
server_key_iv: "d63ce3b6dbb062221b2b1480ca880d12"
server_key_version: "encryptionkey@0"
I20260925 15:19:38.302933 29017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:38.313215 29017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:38.315614 29017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:38.315781 29017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:38.319694 29017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36527
I20260925 15:19:38.319772 29084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36527 every 8 connection(s)
I20260925 15:19:38.320132 29017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:38.321125 29085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:38.323414 29085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8: Bootstrap starting.
I20260925 15:19:38.323968 29085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:38.324252 29085 log.cc:824] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:38.324905 29085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8: No bootstrap required, opened a new log
I20260925 15:19:38.326469 29085 raft_consensus.cc:348] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64837dba74145ff80fc136b52df64c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36527 } }
I20260925 15:19:38.326637 29085 raft_consensus.cc:374] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:38.326681 29085 raft_consensus.cc:729] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64837dba74145ff80fc136b52df64c8, State: Initialized, Role: FOLLOWER
I20260925 15:19:38.326845 29085 consensus_queue.cc:260] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64837dba74145ff80fc136b52df64c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36527 } }
I20260925 15:19:38.326926 29085 raft_consensus.cc:388] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:38.326982 29085 raft_consensus.cc:482] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:38.327041 29085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:38.327587 29085 raft_consensus.cc:504] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64837dba74145ff80fc136b52df64c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36527 } }
I20260925 15:19:38.327741 29085 leader_election.cc:302] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d64837dba74145ff80fc136b52df64c8; no voters:
I20260925 15:19:38.327919 29085 leader_election.cc:288] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:38.328073 29088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:38.328181 29085 sys_catalog.cc:564] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:38.328326 29088 raft_consensus.cc:686] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [term 1 LEADER]: Becoming Leader. State: Replica: d64837dba74145ff80fc136b52df64c8, State: Running, Role: LEADER
I20260925 15:19:38.328498 29088 consensus_queue.cc:237] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64837dba74145ff80fc136b52df64c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36527 } }
I20260925 15:19:38.329175 29090 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d64837dba74145ff80fc136b52df64c8. Latest consensus state: current_term: 1 leader_uuid: "d64837dba74145ff80fc136b52df64c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64837dba74145ff80fc136b52df64c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36527 } } }
I20260925 15:19:38.329313 29090 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:38.329329 29089 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d64837dba74145ff80fc136b52df64c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64837dba74145ff80fc136b52df64c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36527 } } }
I20260925 15:19:38.329411 29089 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:38.329907 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29017
I20260925 15:19:38.330035 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
W20260925 15:19:38.330062 29102 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:38.330137 29102 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:38.330207 29103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:38.330257 8366 external_mini_cluster.cc:1428] Setting key 3ac892228e18bc52be0847dc061a7f2a
I20260925 15:19:38.330422 29103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:38.331979 29103 catalog_manager.cc:1347] Generated new cluster ID: e8fd44d9c5fb43cdb0babfdcc2e9dfa0
I20260925 15:19:38.332029 29103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:38.335583 29107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:38.330775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58589 (local address 127.0.0.1:36527)
0925 15:19:38.331344 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:38.331353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:38.331385 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:19:38.331519 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:38.331527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:38.331599 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:38.331762 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:38.331982 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:38.332909 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:38.333712 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:38.334121 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:38.334245 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:38.334924 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:38.334927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:38.334927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:38.335105 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:38.335135 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:38.335145 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:38.335216 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:38.335324 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:19:38.335406 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":82,"threads_started":1}
I20260925 15:19:38.336474 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:38.344465 29103 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:38.345140 29103 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:38.346312 29107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:38.338072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40153 (local address 127.0.0.1:36527)
0925 15:19:38.338132 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:38.338134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:38.343420 (+ 5286us) server_negotiation.cc:407] Connection header received
0925 15:19:38.343505 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:38.343506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:38.343517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:38.343561 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:38.343709 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:38.344208 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:38.345077 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:38.345282 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:38.345361 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:38.346037 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:38.346038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:38.346039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:38.346043 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:38.346057 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:38.346061 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:38.346113 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:38.346176 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 15:19:38.346213 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:19:38.354413 29103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d64837dba74145ff80fc136b52df64c8: Generated new TSK 0
I20260925 15:19:38.354645 29103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:38.359187 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29017
2026-09-25T15:19:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:38 dist-test-slave-c5d6 krb5kdc[29116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:38 dist-test-slave-c5d6 krb5kdc[29116](info): set up 2 sockets
Sep 25 15:19:38 dist-test-slave-c5d6 krb5kdc[29116](info): commencing operation
krb5kdc: starting...
W20260925 15:19:39.391403 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:39 dist-test-slave-c5d6 krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:39.508368 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:39.587431 29129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:39.587687 29129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:39.587726 29129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:39.588899 29129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:39.588970 29129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:39.588997 29129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:39.589016 29129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:39.589069 29129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:39.590583 29129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:39.590855 29129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:39.591094 29129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:39.595855 29135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:39.595844 29134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:39.595952 29137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:39.595976 29129 server_base.cc:1034] running on GCE node
I20260925 15:19:39.596318 29129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:39.596654 29129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:39.597831 29129 hybrid_clock.cc:648] HybridClock initialized: now 1790349579597826 us; error 34 us; skew 500 ppm
Sep 25 15:19:39 dist-test-slave-c5d6 krb5kdc[29116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:39.605679 29129 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:39.606142 29129 webserver.cc:460] Webserver started at http://127.0.0.1:34257/ using document root <none> and password file <none>
I20260925 15:19:39.606335 29129 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:39.606375 29129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:39.606462 29129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:39.607538 29129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "169e67d8ce42435cb98693c18b959ffa"
format_stamp: "Formatted at 2026-09-25 15:19:39 on dist-test-slave-c5d6"
server_key: "980938249349fc672e11d4d53e0d7443"
server_key_iv: "23f3bcb5da312e1c325f977b8b795d6f"
server_key_version: "encryptionkey@0"
I20260925 15:19:39.607836 29129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "169e67d8ce42435cb98693c18b959ffa"
format_stamp: "Formatted at 2026-09-25 15:19:39 on dist-test-slave-c5d6"
server_key: "980938249349fc672e11d4d53e0d7443"
server_key_iv: "23f3bcb5da312e1c325f977b8b795d6f"
server_key_version: "encryptionkey@0"
I20260925 15:19:39.609130 29129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:39.609866 29144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:39.610128 29129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:39.610201 29129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "169e67d8ce42435cb98693c18b959ffa"
format_stamp: "Formatted at 2026-09-25 15:19:39 on dist-test-slave-c5d6"
server_key: "980938249349fc672e11d4d53e0d7443"
server_key_iv: "23f3bcb5da312e1c325f977b8b795d6f"
server_key_version: "encryptionkey@0"
I20260925 15:19:39.610293 29129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:39.613648 29129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:39.615936 29129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:39.616094 29129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:39.620285 29129 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42485
I20260925 15:19:39.620496 29196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42485 every 8 connection(s)
I20260925 15:19:39.620653 29129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:39.621568 29197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:39.623085 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29129
I20260925 15:19:39.623224 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:39.623449 8366 external_mini_cluster.cc:1428] Setting key b223120eb963d64d043bfeff14275e69
I20260925 15:19:39.624332 29197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa: Bootstrap starting.
I20260925 15:19:39.624962 29197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:39.625281 29197 log.cc:824] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:39.625949 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.623995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59265 (local address 127.0.0.1:42485)
0925 15:19:39.624178 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.624187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.624213 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:19:39.624689 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.624696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.624724 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.624847 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:39.625484 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:39.625507 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:39.625508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:39.625513 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:39.625558 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:39.625573 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:39.625669 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.625704 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.625747 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260925 15:19:39.626099 29197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa: No bootstrap required, opened a new log
I20260925 15:19:39.626860 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:39.628245 29197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169e67d8ce42435cb98693c18b959ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42485 } }
I20260925 15:19:39.628434 29197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:39.628492 29197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169e67d8ce42435cb98693c18b959ffa, State: Initialized, Role: FOLLOWER
I20260925 15:19:39.628633 29197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169e67d8ce42435cb98693c18b959ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42485 } }
I20260925 15:19:39.628710 29197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:39.628734 29197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:39.629060 29197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:39.629345 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:39.628374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58371 (local address 127.0.0.1:42485)
0925 15:19:39.628416 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:39.628418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:39.628466 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:19:39.628526 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:39.628528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:39.628533 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:39.628582 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:39.629131 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:39.629132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:39.629133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:39.629137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:39.629149 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:39.629154 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:39.629199 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:39.629253 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:19:39.629266 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:19:39.629720 29197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169e67d8ce42435cb98693c18b959ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42485 } }
I20260925 15:19:39.629848 29197 leader_election.cc:302] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 169e67d8ce42435cb98693c18b959ffa; no voters:
I20260925 15:19:39.630024 29197 leader_election.cc:288] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:39.630119 29207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:39.630266 29207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [term 1 LEADER]: Becoming Leader. State: Replica: 169e67d8ce42435cb98693c18b959ffa, State: Running, Role: LEADER
I20260925 15:19:39.630311 29197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:39.630388 29207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169e67d8ce42435cb98693c18b959ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42485 } }
I20260925 15:19:39.630751 29208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "169e67d8ce42435cb98693c18b959ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169e67d8ce42435cb98693c18b959ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42485 } } }
I20260925 15:19:39.630857 29208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:39.630751 29209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169e67d8ce42435cb98693c18b959ffa. Latest consensus state: current_term: 1 leader_uuid: "169e67d8ce42435cb98693c18b959ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "169e67d8ce42435cb98693c18b959ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42485 } } }
I20260925 15:19:39.631081 29214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:39.631141 29209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:39.631673 29214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:39.636296 29214 catalog_manager.cc:1347] Generated new cluster ID: edbc92ff07224835aca0d9928b81d529
I20260925 15:19:39.636373 29214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:39.651949 29214 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:39.652145 29214 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:39.654776 29214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 169e67d8ce42435cb98693c18b959ffa: Generated new TSK 0
I20260925 15:19:39.654995 29214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:39.661258 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29129
2026-09-25T15:19:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:39 dist-test-slave-c5d6 krb5kdc[29228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:39 dist-test-slave-c5d6 krb5kdc[29228](info): set up 2 sockets
Sep 25 15:19:39 dist-test-slave-c5d6 krb5kdc[29228](info): commencing operation
krb5kdc: starting...
W20260925 15:19:40.687363 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:40 dist-test-slave-c5d6 krb5kdc[29228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:40.803923 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:40.884446 29241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:40.884649 29241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:40.884671 29241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:40.885777 29241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:40.885815 29241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:40.885830 29241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:40.885842 29241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:40.885855 29241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:40.887259 29241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:40.887445 29241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:40.887674 29241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:40.892406 29249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:40.892460 29247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:40.892423 29246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:40.892617 29241 server_base.cc:1034] running on GCE node
I20260925 15:19:40.892848 29241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:40.893157 29241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:40.894313 29241 hybrid_clock.cc:648] HybridClock initialized: now 1790349580894309 us; error 38 us; skew 500 ppm
Sep 25 15:19:40 dist-test-slave-c5d6 krb5kdc[29228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:40.896739 29241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:40.897148 29241 webserver.cc:460] Webserver started at http://127.0.0.1:43537/ using document root <none> and password file <none>
I20260925 15:19:40.897369 29241 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:40.897436 29241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:40.897593 29241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:40.898627 29241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "617a591a92154355b4ca80502c114cb3"
format_stamp: "Formatted at 2026-09-25 15:19:40 on dist-test-slave-c5d6"
server_key: "02b1e849e82089e17380c0cac2194d50"
server_key_iv: "8a012321d7ad7a43ba458e9452125ac8"
server_key_version: "encryptionkey@0"
I20260925 15:19:40.898993 29241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "617a591a92154355b4ca80502c114cb3"
format_stamp: "Formatted at 2026-09-25 15:19:40 on dist-test-slave-c5d6"
server_key: "02b1e849e82089e17380c0cac2194d50"
server_key_iv: "8a012321d7ad7a43ba458e9452125ac8"
server_key_version: "encryptionkey@0"
I20260925 15:19:40.900338 29241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 15:19:40.901052 29256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:40.901222 29241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:40.901317 29241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "617a591a92154355b4ca80502c114cb3"
format_stamp: "Formatted at 2026-09-25 15:19:40 on dist-test-slave-c5d6"
server_key: "02b1e849e82089e17380c0cac2194d50"
server_key_iv: "8a012321d7ad7a43ba458e9452125ac8"
server_key_version: "encryptionkey@0"
I20260925 15:19:40.901403 29241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:40.921162 29241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:40.922925 29241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:40.923112 29241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:40.926858 29241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37473
I20260925 15:19:40.926916 29308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37473 every 8 connection(s)
I20260925 15:19:40.927287 29241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:40.928190 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29241
I20260925 15:19:40.928313 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:40.928419 29309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:40.928553 8366 external_mini_cluster.cc:1428] Setting key 289bc263c20aa3cb59aaeae0e833677a
I20260925 15:19:40.931142 29309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3: Bootstrap starting.
I20260925 15:19:40.931743 29309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:40.932036 29309 log.cc:824] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:40.932665 29309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3: No bootstrap required, opened a new log
I20260925 15:19:40.934731 29309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617a591a92154355b4ca80502c114cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37473 } }
I20260925 15:19:40.934909 29309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:40.934943 29309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617a591a92154355b4ca80502c114cb3, State: Initialized, Role: FOLLOWER
Sep 25 15:19:40 dist-test-slave-c5d6 krb5kdc[29228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:19:40.935079 29309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617a591a92154355b4ca80502c114cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37473 } }
I20260925 15:19:40.935169 29309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:40.935258 29309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:40.935316 29309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:40.935873 29309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617a591a92154355b4ca80502c114cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37473 } }
I20260925 15:19:40.935990 29309 leader_election.cc:302] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617a591a92154355b4ca80502c114cb3; no voters:
I20260925 15:19:40.936170 29309 leader_election.cc:288] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:40.936223 29314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:40.936378 29314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 617a591a92154355b4ca80502c114cb3, State: Running, Role: LEADER
I20260925 15:19:40.936447 29309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:40.936739 29314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617a591a92154355b4ca80502c114cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37473 } }
I20260925 15:19:40.937115 29315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617a591a92154355b4ca80502c114cb3. Latest consensus state: current_term: 1 leader_uuid: "617a591a92154355b4ca80502c114cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617a591a92154355b4ca80502c114cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37473 } } }
I20260925 15:19:40.937196 29315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:40.937413 29316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "617a591a92154355b4ca80502c114cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617a591a92154355b4ca80502c114cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37473 } } }
I20260925 15:19:40.937700 29316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:40.937844 29322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:40.938314 29322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:40.939396 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:40.929065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:54427 (local address 127.0.0.1:37473)
0925 15:19:40.929242 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:40.929252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:40.929285 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:40.929814 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:40.929821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:40.929869 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:40.930031 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:40.930638 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:40.931618 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:40.932499 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:40.932945 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:40.933080 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:40.935632 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:40.935635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:40.935636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:40.935648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:40.937664 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:40.938186 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:40.938189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:40.938189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:40.938292 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:40.938538 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:40.938539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:40.938539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:40.938783 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:40.938949 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:40.939056 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:19:40.939162 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260925 15:19:40.940354 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:40.943286 29322 catalog_manager.cc:1347] Generated new cluster ID: 61fc92e8f6874ff0a670146aee0a7411
I20260925 15:19:40.943342 29322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:40.947861 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:40.941704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:34707 (local address 127.0.0.1:37473)
0925 15:19:40.941760 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:40.941762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:40.941880 (+ 118us) server_negotiation.cc:407] Connection header received
0925 15:19:40.941938 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:40.941939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:40.941949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:40.942040 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:40.942474 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:40.943207 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:40.943953 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:40.944263 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:40.944331 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:40.945427 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:40.945429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:40.945429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:40.945435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:40.946734 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:40.947062 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:40.947063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:40.947063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:40.947116 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:40.947358 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:40.947358 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:40.947359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:40.947473 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:40.947565 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:40.947724 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:19:40.947781 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:19:40.962175 29322 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:40.962692 29322 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:40.968645 29322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 617a591a92154355b4ca80502c114cb3: Generated new TSK 0
I20260925 15:19:40.968835 29322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:40.980170 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29241
W20260925 15:19:40.984411 29226 connection.cc:535] client connection to 127.0.0.1:37473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37473): Connection reset by peer (error 104)
2026-09-25T15:19:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:41 dist-test-slave-c5d6 krb5kdc[29340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:41 dist-test-slave-c5d6 krb5kdc[29340](info): set up 2 sockets
Sep 25 15:19:41 dist-test-slave-c5d6 krb5kdc[29340](info): commencing operation
krb5kdc: starting...
W20260925 15:19:42.003393 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:42 dist-test-slave-c5d6 krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:42.121390 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:42.201932 29353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:42.202147 29353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:42.202185 29353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:42.203383 29353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:42.203444 29353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:42.203467 29353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:42.203495 29353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:42.203524 29353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:42.205039 29353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:42.205286 29353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:42.205507 29353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:42.210296 29359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:42.210297 29358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:42.210297 29361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:42.210382 29353 server_base.cc:1034] running on GCE node
I20260925 15:19:42.210727 29353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:42.210963 29353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:42.212144 29353 hybrid_clock.cc:648] HybridClock initialized: now 1790349582212129 us; error 28 us; skew 500 ppm
Sep 25 15:19:42 dist-test-slave-c5d6 krb5kdc[29340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:42.214483 29353 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:42.214913 29353 webserver.cc:460] Webserver started at http://127.0.0.1:42325/ using document root <none> and password file <none>
I20260925 15:19:42.215112 29353 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:42.215153 29353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:42.215296 29353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:42.216308 29353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "bc98f3344dd0480dabaf7d75394d5922"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-c5d6"
server_key: "6c0a1fc785fb56920037f67340981067"
server_key_iv: "ba136392682ba9c945d775994c0fc01c"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.216590 29353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "bc98f3344dd0480dabaf7d75394d5922"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-c5d6"
server_key: "6c0a1fc785fb56920037f67340981067"
server_key_iv: "ba136392682ba9c945d775994c0fc01c"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.217780 29353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:42.218444 29368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:42.218611 29353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:42.218696 29353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "bc98f3344dd0480dabaf7d75394d5922"
format_stamp: "Formatted at 2026-09-25 15:19:42 on dist-test-slave-c5d6"
server_key: "6c0a1fc785fb56920037f67340981067"
server_key_iv: "ba136392682ba9c945d775994c0fc01c"
server_key_version: "encryptionkey@0"
I20260925 15:19:42.218786 29353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:42.237843 29353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:42.239928 29353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:42.240091 29353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:42.244414 29353 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41531
I20260925 15:19:42.244489 29420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41531 every 8 connection(s)
I20260925 15:19:42.244803 29353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:42.245802 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:42.245858 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29353
I20260925 15:19:42.245975 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:42.246189 8366 external_mini_cluster.cc:1428] Setting key 462035edafd17cb82a1ddc596ab23a4d
I20260925 15:19:42.249431 29421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922: Bootstrap starting.
I20260925 15:19:42.250037 29421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:42.250331 29421 log.cc:824] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:42.251016 29421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922: No bootstrap required, opened a new log
I20260925 15:19:42.252300 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.246729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:52705 (local address 127.0.0.1:41531)
0925 15:19:42.246926 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:42.246936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:42.246969 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:42.247640 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:42.247649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:42.247718 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:42.247869 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:42.248070 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.249409 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.250238 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.250699 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.250815 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.251598 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:42.251602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:42.251603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:42.251769 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:42.251796 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:42.251807 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:42.251899 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:42.251979 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:19:42.252005 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260925 15:19:42.252596 29421 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98f3344dd0480dabaf7d75394d5922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260925 15:19:42.252750 29421 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:42.252795 29421 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc98f3344dd0480dabaf7d75394d5922, State: Initialized, Role: FOLLOWER
I20260925 15:19:42.252919 29421 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98f3344dd0480dabaf7d75394d5922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260925 15:19:42.253010 29421 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:42.253052 29421 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:42.253100 29421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:42.253201 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:42.253665 29421 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98f3344dd0480dabaf7d75394d5922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260925 15:19:42.253782 29421 leader_election.cc:302] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc98f3344dd0480dabaf7d75394d5922; no voters:
I20260925 15:19:42.253978 29421 leader_election.cc:288] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:42.254022 29426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:42.254211 29426 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [term 1 LEADER]: Becoming Leader. State: Replica: bc98f3344dd0480dabaf7d75394d5922, State: Running, Role: LEADER
I20260925 15:19:42.254395 29426 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98f3344dd0480dabaf7d75394d5922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260925 15:19:42.254777 29426 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc98f3344dd0480dabaf7d75394d5922. Latest consensus state: current_term: 1 leader_uuid: "bc98f3344dd0480dabaf7d75394d5922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98f3344dd0480dabaf7d75394d5922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } } }
I20260925 15:19:42.254787 29427 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc98f3344dd0480dabaf7d75394d5922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc98f3344dd0480dabaf7d75394d5922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } } }
I20260925 15:19:42.254884 29426 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:42.254909 29427 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:42.254232 29421 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:42.255125 29435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:42.255658 29435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:42.257647 29435 catalog_manager.cc:1347] Generated new cluster ID: ddecf33b87e94bffabd3ac4495bae447
I20260925 15:19:42.257704 29435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:42.259971 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:42.254662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:53099 (local address 127.0.0.1:41531)
0925 15:19:42.254694 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:42.254696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:42.254816 (+ 120us) server_negotiation.cc:407] Connection header received
0925 15:19:42.254909 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:42.254910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:42.254921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:42.254983 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:42.255508 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.256398 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.257302 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:42.257569 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:42.257632 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:42.259724 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:42.259726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:42.259726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:42.259730 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:42.259743 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:42.259746 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:42.259799 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:42.259884 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 15:19:42.259893 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:19:42.274448 29435 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:42.275017 29435 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:42.277689 29435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc98f3344dd0480dabaf7d75394d5922: Generated new TSK 0
I20260925 15:19:42.277877 29435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:42.285637 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29353
W20260925 15:19:42.289896 29338 connection.cc:535] client connection to 127.0.0.1:41531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41531): Connection reset by peer (error 104)
2026-09-25T15:19:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:42 dist-test-slave-c5d6 krb5kdc[29452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:42 dist-test-slave-c5d6 krb5kdc[29452](info): set up 2 sockets
Sep 25 15:19:42 dist-test-slave-c5d6 krb5kdc[29452](info): commencing operation
krb5kdc: starting...
W20260925 15:19:43.303463 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:43 dist-test-slave-c5d6 krb5kdc[29452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:43.422497 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:43.508513 29465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:43.508769 29465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:43.508811 29465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:43.509994 29465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:43.510072 29465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:43.510099 29465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:43.510131 29465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:43.510190 29465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:43.511722 29465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:43.512022 29465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:43.512261 29465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:19:43.517485 29465 server_base.cc:1034] running on GCE node
W20260925 15:19:43.517398 29471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:43.517441 29473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:43.517441 29470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:43.517939 29465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:43.518299 29465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:43.519464 29465 hybrid_clock.cc:648] HybridClock initialized: now 1790349583519448 us; error 34 us; skew 500 ppm
Sep 25 15:19:43 dist-test-slave-c5d6 krb5kdc[29452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:43.521907 29465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:43.522464 29465 webserver.cc:460] Webserver started at http://127.0.0.1:44063/ using document root <none> and password file <none>
I20260925 15:19:43.522694 29465 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:43.522760 29465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:43.522912 29465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:43.524061 29465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "c73bd200b83340728ec5dea792a85ea1"
format_stamp: "Formatted at 2026-09-25 15:19:43 on dist-test-slave-c5d6"
server_key: "68009076941f023e2f8ea89ab5aa1b65"
server_key_iv: "9ce52a4b797007ecfc9baac7f6b14414"
server_key_version: "encryptionkey@0"
I20260925 15:19:43.524420 29465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "c73bd200b83340728ec5dea792a85ea1"
format_stamp: "Formatted at 2026-09-25 15:19:43 on dist-test-slave-c5d6"
server_key: "68009076941f023e2f8ea89ab5aa1b65"
server_key_iv: "9ce52a4b797007ecfc9baac7f6b14414"
server_key_version: "encryptionkey@0"
I20260925 15:19:43.525751 29465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:19:43.526506 29480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:43.526701 29465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:43.526806 29465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "c73bd200b83340728ec5dea792a85ea1"
format_stamp: "Formatted at 2026-09-25 15:19:43 on dist-test-slave-c5d6"
server_key: "68009076941f023e2f8ea89ab5aa1b65"
server_key_iv: "9ce52a4b797007ecfc9baac7f6b14414"
server_key_version: "encryptionkey@0"
I20260925 15:19:43.526881 29465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:43.530180 29465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:43.532255 29465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:43.532418 29465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:43.536475 29465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41515
I20260925 15:19:43.536502 29532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41515 every 8 connection(s)
I20260925 15:19:43.536863 29465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:43.537130 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29465
I20260925 15:19:43.537258 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:43.537456 8366 external_mini_cluster.cc:1428] Setting key 422aba5cbe35281405a482b09f80314f
I20260925 15:19:43.538076 29533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:43.540117 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:43.537986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:58979 (local address 127.0.0.1:41515)
0925 15:19:43.538194 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:43.538204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:43.538237 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:43.538824 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:43.538833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:43.538878 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:43.539041 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:43.539683 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:43.539687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:43.539687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:43.539692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:43.539734 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:43.539748 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:43.539824 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:43.539876 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 15:19:43.539918 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260925 15:19:43.541153 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:43.541441 29533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1: Bootstrap starting.
I20260925 15:19:43.542164 29533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:43.542526 29533 log.cc:824] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:43.543321 29533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1: No bootstrap required, opened a new log
I20260925 15:19:43.543955 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:43.542806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:50921 (local address 127.0.0.1:41515)
0925 15:19:43.542839 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:43.542841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:43.542973 (+ 132us) server_negotiation.cc:407] Connection header received
0925 15:19:43.543041 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:43.543042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:43.543047 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:43.543114 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:43.543725 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:43.543726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:43.543726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:43.543730 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:43.543743 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:43.543747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:43.543800 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:43.543860 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:19:43.543874 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 15:19:43.545240 29533 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73bd200b83340728ec5dea792a85ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } }
I20260925 15:19:43.545408 29533 raft_consensus.cc:374] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:43.545459 29533 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73bd200b83340728ec5dea792a85ea1, State: Initialized, Role: FOLLOWER
I20260925 15:19:43.545585 29533 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73bd200b83340728ec5dea792a85ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } }
I20260925 15:19:43.545693 29533 raft_consensus.cc:388] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:43.545737 29533 raft_consensus.cc:482] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:43.545789 29533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:43.546308 29533 raft_consensus.cc:504] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73bd200b83340728ec5dea792a85ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } }
I20260925 15:19:43.546429 29533 leader_election.cc:302] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c73bd200b83340728ec5dea792a85ea1; no voters:
I20260925 15:19:43.546620 29533 leader_election.cc:288] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:43.546733 29543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:43.546893 29543 raft_consensus.cc:686] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [term 1 LEADER]: Becoming Leader. State: Replica: c73bd200b83340728ec5dea792a85ea1, State: Running, Role: LEADER
I20260925 15:19:43.546929 29533 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:43.547037 29543 consensus_queue.cc:237] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73bd200b83340728ec5dea792a85ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } }
I20260925 15:19:43.547504 29545 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c73bd200b83340728ec5dea792a85ea1. Latest consensus state: current_term: 1 leader_uuid: "c73bd200b83340728ec5dea792a85ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73bd200b83340728ec5dea792a85ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } } }
I20260925 15:19:43.547636 29545 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:43.547490 29544 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c73bd200b83340728ec5dea792a85ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73bd200b83340728ec5dea792a85ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41515 } } }
I20260925 15:19:43.548061 29544 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:43.548236 29552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:43.548791 29552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:43.550249 29552 catalog_manager.cc:1347] Generated new cluster ID: 6e9f02f605a5478582297230be341c88
I20260925 15:19:43.550297 29552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:43.569674 29552 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:43.569897 29552 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:43.574386 29552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c73bd200b83340728ec5dea792a85ea1: Generated new TSK 0
I20260925 15:19:43.574596 29552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:43.578969 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29465
2026-09-25T15:19:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:43 dist-test-slave-c5d6 krb5kdc[29564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:43 dist-test-slave-c5d6 krb5kdc[29564](info): set up 2 sockets
Sep 25 15:19:43 dist-test-slave-c5d6 krb5kdc[29564](info): commencing operation
krb5kdc: starting...
W20260925 15:19:44.623513 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:44 dist-test-slave-c5d6 krb5kdc[29564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:44.741632 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:44.822650 29577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:44.822923 29577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:44.822978 29577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:44.824786 29577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:44.824853 29577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:44.824877 29577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:44.824908 29577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:44.824932 29577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:44.826915 29577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:44.827183 29577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:44.827446 29577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:44.832131 29583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:44.832154 29582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:44.832266 29577 server_base.cc:1034] running on GCE node
W20260925 15:19:44.832176 29585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:44.832548 29577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:44.832873 29577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:44.834062 29577 hybrid_clock.cc:648] HybridClock initialized: now 1790349584834044 us; error 24 us; skew 500 ppm
Sep 25 15:19:44 dist-test-slave-c5d6 krb5kdc[29564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:44.845240 29577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:44.845734 29577 webserver.cc:460] Webserver started at http://127.0.0.1:43247/ using document root <none> and password file <none>
I20260925 15:19:44.845965 29577 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:44.846035 29577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:44.846184 29577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:44.847306 29577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "3e5118cd512d4efcbaa327f6cdeb1a51"
format_stamp: "Formatted at 2026-09-25 15:19:44 on dist-test-slave-c5d6"
server_key: "b6aa1f7a068787563c35a22433c26383"
server_key_iv: "1fe91862a2e6df0e127a79961ab451bb"
server_key_version: "encryptionkey@0"
I20260925 15:19:44.847669 29577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "3e5118cd512d4efcbaa327f6cdeb1a51"
format_stamp: "Formatted at 2026-09-25 15:19:44 on dist-test-slave-c5d6"
server_key: "b6aa1f7a068787563c35a22433c26383"
server_key_iv: "1fe91862a2e6df0e127a79961ab451bb"
server_key_version: "encryptionkey@0"
I20260925 15:19:44.848990 29577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:44.849668 29592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:44.849830 29577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:44.849931 29577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "3e5118cd512d4efcbaa327f6cdeb1a51"
format_stamp: "Formatted at 2026-09-25 15:19:44 on dist-test-slave-c5d6"
server_key: "b6aa1f7a068787563c35a22433c26383"
server_key_iv: "1fe91862a2e6df0e127a79961ab451bb"
server_key_version: "encryptionkey@0"
I20260925 15:19:44.850001 29577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:44.857540 29577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:44.859724 29577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:44.859876 29577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:44.863862 29577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36395
I20260925 15:19:44.863958 29644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36395 every 8 connection(s)
I20260925 15:19:44.864260 29577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:44.865303 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:44.865912 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29577
I20260925 15:19:44.866040 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:44.866250 8366 external_mini_cluster.cc:1428] Setting key 9c8035502cadad7c161f880e19e849a9
I20260925 15:19:44.868609 29645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51: Bootstrap starting.
I20260925 15:19:44.869226 29645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:44.869536 29645 log.cc:824] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:44.870288 29645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51: No bootstrap required, opened a new log
I20260925 15:19:44.872967 29645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5118cd512d4efcbaa327f6cdeb1a51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } }
I20260925 15:19:44.873102 29645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:44.873126 29645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5118cd512d4efcbaa327f6cdeb1a51, State: Initialized, Role: FOLLOWER
I20260925 15:19:44.873273 29645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5118cd512d4efcbaa327f6cdeb1a51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } }
I20260925 15:19:44.873349 29645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:44.873395 29645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 25 15:19:44 dist-test-slave-c5d6 krb5kdc[29564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:19:44.873448 29645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:44.873981 29645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5118cd512d4efcbaa327f6cdeb1a51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } }
I20260925 15:19:44.874105 29645 leader_election.cc:302] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e5118cd512d4efcbaa327f6cdeb1a51; no voters:
I20260925 15:19:44.874305 29645 leader_election.cc:288] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:44.874339 29650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:44.874509 29650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [term 1 LEADER]: Becoming Leader. State: Replica: 3e5118cd512d4efcbaa327f6cdeb1a51, State: Running, Role: LEADER
I20260925 15:19:44.874560 29645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:44.874660 29650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5118cd512d4efcbaa327f6cdeb1a51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } }
I20260925 15:19:44.875105 29651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e5118cd512d4efcbaa327f6cdeb1a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5118cd512d4efcbaa327f6cdeb1a51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } } }
I20260925 15:19:44.875238 29651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:44.875481 29658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:44.875660 29652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e5118cd512d4efcbaa327f6cdeb1a51. Latest consensus state: current_term: 1 leader_uuid: "3e5118cd512d4efcbaa327f6cdeb1a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e5118cd512d4efcbaa327f6cdeb1a51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } } }
I20260925 15:19:44.875828 29652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:44.876016 29658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:44.877696 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.866783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:36963 (local address 127.0.0.1:36395)
0925 15:19:44.866933 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.866942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.866997 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:19:44.867459 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.867466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.867515 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.867625 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:44.868523 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.869467 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.870537 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.870937 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.871021 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.873886 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:44.873889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:44.873890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:44.873901 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:44.876036 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:44.876495 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:44.876498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:44.876499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:44.876598 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:44.876845 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:44.876846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:44.876846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:44.877097 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:44.877301 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.877387 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.877501 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260925 15:19:44.878703 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:44.880470 29666 mvcc.cc:205] Tried to move back new op lower bound from 7333271899656294400 to 7333271899647332352. Current Snapshot: MvccSnapshot[applied={T|T < 7333271899656294400}]
I20260925 15:19:44.880722 29658 catalog_manager.cc:1347] Generated new cluster ID: 7b39a64c9df64fcaaa020cc27381ef48
I20260925 15:19:44.880776 29658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:44.886737 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:44.880258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:46183 (local address 127.0.0.1:36395)
0925 15:19:44.880288 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:44.880290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:44.880410 (+ 120us) server_negotiation.cc:407] Connection header received
0925 15:19:44.880564 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:44.880565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:44.880574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:44.880603 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:44.881222 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.881938 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.882669 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:44.882913 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:44.883001 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:44.884179 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:44.884180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:44.884180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:44.884187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:44.885592 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:44.885958 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:44.885959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:44.885959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:44.886018 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:44.886245 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:44.886246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:44.886247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:44.886356 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:44.886463 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:44.886584 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:19:44.886643 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 15:19:44.899164 29658 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:44.899770 29658 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:44.906284 29658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e5118cd512d4efcbaa327f6cdeb1a51: Generated new TSK 0
I20260925 15:19:44.906471 29658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:44.913213 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29577
W20260925 15:19:44.917752 29562 connection.cc:535] client connection to 127.0.0.1:36395 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36395): Connection reset by peer (error 104)
2026-09-25T15:19:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:44 dist-test-slave-c5d6 krb5kdc[29676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:44 dist-test-slave-c5d6 krb5kdc[29676](info): set up 2 sockets
Sep 25 15:19:44 dist-test-slave-c5d6 krb5kdc[29676](info): commencing operation
krb5kdc: starting...
W20260925 15:19:45.947479 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:46 dist-test-slave-c5d6 krb5kdc[29676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:46.065591 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:46.144371 29689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:46.144618 29689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:46.144657 29689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:46.145754 29689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:46.145816 29689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:46.145839 29689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:46.145859 29689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:46.145885 29689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:46.147358 29689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:46.147697 29689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:46.147994 29689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:46.152774 29697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:46.152916 29694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:46.152978 29695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:46.153295 29689 server_base.cc:1034] running on GCE node
I20260925 15:19:46.153545 29689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:46.153828 29689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:46.154997 29689 hybrid_clock.cc:648] HybridClock initialized: now 1790349586155003 us; error 44 us; skew 500 ppm
Sep 25 15:19:46 dist-test-slave-c5d6 krb5kdc[29676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:46.157539 29689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:46.158020 29689 webserver.cc:460] Webserver started at http://127.0.0.1:42419/ using document root <none> and password file <none>
I20260925 15:19:46.158255 29689 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:46.158322 29689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:46.158473 29689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:46.159564 29689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "8d9dd856544c4dbdb5bb9314836b034d"
format_stamp: "Formatted at 2026-09-25 15:19:46 on dist-test-slave-c5d6"
server_key: "f192030ad67cb081cadacfed1c5dc66f"
server_key_iv: "823409f09080869fe1fcfbdb1d0187e4"
server_key_version: "encryptionkey@0"
I20260925 15:19:46.159909 29689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "8d9dd856544c4dbdb5bb9314836b034d"
format_stamp: "Formatted at 2026-09-25 15:19:46 on dist-test-slave-c5d6"
server_key: "f192030ad67cb081cadacfed1c5dc66f"
server_key_iv: "823409f09080869fe1fcfbdb1d0187e4"
server_key_version: "encryptionkey@0"
I20260925 15:19:46.161201 29689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:46.161902 29704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:46.162101 29689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:46.162204 29689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "8d9dd856544c4dbdb5bb9314836b034d"
format_stamp: "Formatted at 2026-09-25 15:19:46 on dist-test-slave-c5d6"
server_key: "f192030ad67cb081cadacfed1c5dc66f"
server_key_iv: "823409f09080869fe1fcfbdb1d0187e4"
server_key_version: "encryptionkey@0"
I20260925 15:19:46.162277 29689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:46.177708 29689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:46.180053 29689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:46.180210 29689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:46.184216 29689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37757
I20260925 15:19:46.184253 29756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37757 every 8 connection(s)
I20260925 15:19:46.184587 29689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:46.185421 29757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:46.187647 29757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d: Bootstrap starting.
I20260925 15:19:46.188215 29757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:46.188500 29757 log.cc:824] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:46.189193 29757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d: No bootstrap required, opened a new log
I20260925 15:19:46.190070 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29689
I20260925 15:19:46.190196 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:46.190378 8366 external_mini_cluster.cc:1428] Setting key dbb82920fc569aabe0f0e5c73677ec45
I20260925 15:19:46.191202 29757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9dd856544c4dbdb5bb9314836b034d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37757 } }
I20260925 15:19:46.191390 29757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:46.191424 29757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d9dd856544c4dbdb5bb9314836b034d, State: Initialized, Role: FOLLOWER
I20260925 15:19:46.191552 29757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9dd856544c4dbdb5bb9314836b034d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37757 } }
I20260925 15:19:46.191637 29757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:46.191671 29757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:46.191721 29757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:46.192271 29757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9dd856544c4dbdb5bb9314836b034d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37757 } }
I20260925 15:19:46.192358 29757 leader_election.cc:302] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d9dd856544c4dbdb5bb9314836b034d; no voters:
I20260925 15:19:46.192569 29757 leader_election.cc:288] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:46.192601 29762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:46.192700 29762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [term 1 LEADER]: Becoming Leader. State: Replica: 8d9dd856544c4dbdb5bb9314836b034d, State: Running, Role: LEADER
I20260925 15:19:46.192826 29757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:46.192840 29762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9dd856544c4dbdb5bb9314836b034d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37757 } }
I20260925 15:19:46.193249 29762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d9dd856544c4dbdb5bb9314836b034d. Latest consensus state: current_term: 1 leader_uuid: "8d9dd856544c4dbdb5bb9314836b034d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9dd856544c4dbdb5bb9314836b034d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37757 } } }
I20260925 15:19:46.193257 29763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d9dd856544c4dbdb5bb9314836b034d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d9dd856544c4dbdb5bb9314836b034d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37757 } } }
I20260925 15:19:46.193411 29762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:46.193420 29763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:46.193668 29771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:46.195784 29771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:46.197185 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:46.190943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:46983 (local address 127.0.0.1:37757)
0925 15:19:46.191146 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:46.191156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:46.191189 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:19:46.191326 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:46.191333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:46.191391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:46.191546 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:46.191874 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:46.193687 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:46.194954 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:46.195533 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:46.195961 (+ 428us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:46.196404 (+ 443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:46.196408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:46.196408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:46.196646 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:46.196683 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:46.196698 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:46.196787 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:46.196945 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:19:46.196974 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260925 15:19:46.197434 29771 catalog_manager.cc:1347] Generated new cluster ID: f677d643e1514dd295c95c51fb65d3bf
I20260925 15:19:46.197511 29771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:46.198002 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:46.202780 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:46.199516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:55535 (local address 127.0.0.1:37757)
0925 15:19:46.199547 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:46.199549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:46.199557 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:19:46.199579 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:46.199580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:46.199599 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:46.199665 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:46.199763 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:46.200501 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:46.201476 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:46.201796 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:46.201868 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:46.202530 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:46.202531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:46.202532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:46.202536 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:46.202550 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:46.202554 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:46.202612 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:46.202690 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:19:46.202703 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":16}
I20260925 15:19:46.209923 29771 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:46.210611 29771 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:46.214114 29771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d9dd856544c4dbdb5bb9314836b034d: Generated new TSK 0
I20260925 15:19:46.214274 29771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:46.219228 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29689
W20260925 15:19:46.223419 29674 connection.cc:535] client connection to 127.0.0.1:37757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37757): Connection reset by peer (error 104)
2026-09-25T15:19:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:46 dist-test-slave-c5d6 krb5kdc[29788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:46 dist-test-slave-c5d6 krb5kdc[29788](info): set up 2 sockets
Sep 25 15:19:46 dist-test-slave-c5d6 krb5kdc[29788](info): commencing operation
krb5kdc: starting...
W20260925 15:19:47.275503 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:47 dist-test-slave-c5d6 krb5kdc[29788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:19:47.393537 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:47.476977 29801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:47.477167 29801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:47.477190 29801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:47.478353 29801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:47.478394 29801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:47.478410 29801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:47.478426 29801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:47.478439 29801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:47.479878 29801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:47.480074 29801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:47.480232 29801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:47.484797 29807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:47.484798 29809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:47.484905 29806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:47.485081 29801 server_base.cc:1034] running on GCE node
I20260925 15:19:47.485364 29801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:47.485754 29801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:47.486969 29801 hybrid_clock.cc:648] HybridClock initialized: now 1790349587486947 us; error 32 us; skew 500 ppm
Sep 25 15:19:47 dist-test-slave-c5d6 krb5kdc[29788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:47.489619 29801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:19:47.490041 29801 webserver.cc:460] Webserver started at http://127.0.0.1:34657/ using document root <none> and password file <none>
I20260925 15:19:47.490217 29801 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:47.490254 29801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:47.490338 29801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:47.491403 29801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "44e037b794ef4ad9a631d645c46e11ff"
format_stamp: "Formatted at 2026-09-25 15:19:47 on dist-test-slave-c5d6"
server_key: "a9c41ab4ee3c1a15c077b98ee8308328"
server_key_iv: "2b191ae9d5984470289d8fb250aa9ea6"
server_key_version: "encryptionkey@0"
I20260925 15:19:47.491704 29801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "44e037b794ef4ad9a631d645c46e11ff"
format_stamp: "Formatted at 2026-09-25 15:19:47 on dist-test-slave-c5d6"
server_key: "a9c41ab4ee3c1a15c077b98ee8308328"
server_key_iv: "2b191ae9d5984470289d8fb250aa9ea6"
server_key_version: "encryptionkey@0"
I20260925 15:19:47.492955 29801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:47.493738 29816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:47.493943 29801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:47.494028 29801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "44e037b794ef4ad9a631d645c46e11ff"
format_stamp: "Formatted at 2026-09-25 15:19:47 on dist-test-slave-c5d6"
server_key: "a9c41ab4ee3c1a15c077b98ee8308328"
server_key_iv: "2b191ae9d5984470289d8fb250aa9ea6"
server_key_version: "encryptionkey@0"
I20260925 15:19:47.494081 29801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:47.497802 29801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:47.499715 29801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:47.499817 29801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:47.503446 29801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45861
I20260925 15:19:47.503638 29868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45861 every 8 connection(s)
I20260925 15:19:47.503762 29801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:47.504662 29869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:47.506661 29869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff: Bootstrap starting.
I20260925 15:19:47.507190 29869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:47.507543 29869 log.cc:824] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:47.508018 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29801
I20260925 15:19:47.508153 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:47.508329 8366 external_mini_cluster.cc:1428] Setting key 83ee309ec416303fea5d93a4c21aa902
I20260925 15:19:47.508388 29869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff: No bootstrap required, opened a new log
I20260925 15:19:47.510648 29869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e037b794ef4ad9a631d645c46e11ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260925 15:19:47.510872 29869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:47.510937 29869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e037b794ef4ad9a631d645c46e11ff, State: Initialized, Role: FOLLOWER
I20260925 15:19:47.511066 29869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e037b794ef4ad9a631d645c46e11ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260925 15:19:47.511164 29869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:47.511219 29869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:47.511271 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:47.508843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:40431 (local address 127.0.0.1:45861)
0925 15:19:47.509093 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:47.509103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:47.509135 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:19:47.509783 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:47.509790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:47.509824 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:47.509966 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:47.510600 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:47.510604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:47.510604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:47.510609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:47.510650 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:47.510664 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:47.510734 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:47.510786 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 15:19:47.510827 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":67,"threads_started":1}
I20260925 15:19:47.511296 29869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:47.512065 29869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e037b794ef4ad9a631d645c46e11ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260925 15:19:47.512176 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:47.512174 29869 leader_election.cc:302] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44e037b794ef4ad9a631d645c46e11ff; no voters:
I20260925 15:19:47.512324 29869 leader_election.cc:288] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:47.512348 29874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:47.512457 29874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [term 1 LEADER]: Becoming Leader. State: Replica: 44e037b794ef4ad9a631d645c46e11ff, State: Running, Role: LEADER
I20260925 15:19:47.512576 29869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:47.512583 29874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e037b794ef4ad9a631d645c46e11ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260925 15:19:47.513010 29874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44e037b794ef4ad9a631d645c46e11ff. Latest consensus state: current_term: 1 leader_uuid: "44e037b794ef4ad9a631d645c46e11ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e037b794ef4ad9a631d645c46e11ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } } }
I20260925 15:19:47.513243 29874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:47.513165 29875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44e037b794ef4ad9a631d645c46e11ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44e037b794ef4ad9a631d645c46e11ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } } }
I20260925 15:19:47.513644 29875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:47.513758 29887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:47.514273 29887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:47.515498 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:47.514286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:41923 (local address 127.0.0.1:45861)
0925 15:19:47.514323 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:47.514325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:47.514488 (+ 163us) server_negotiation.cc:407] Connection header received
0925 15:19:47.514578 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:47.514579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:47.514583 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:47.514611 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:47.515282 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:47.515283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:47.515283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:47.515287 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:47.515300 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:47.515304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:47.515367 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:47.515431 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:19:47.515443 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:19:47.516053 29887 catalog_manager.cc:1347] Generated new cluster ID: d94352d9aff8416ca82f04fe02013ee3
I20260925 15:19:47.516099 29887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:47.529404 29887 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:47.529560 29887 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:47.534305 29887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44e037b794ef4ad9a631d645c46e11ff: Generated new TSK 0
I20260925 15:19:47.534463 29887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:47.540099 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29801
2026-09-25T15:19:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:47 dist-test-slave-c5d6 krb5kdc[29900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:47 dist-test-slave-c5d6 krb5kdc[29900](info): set up 2 sockets
Sep 25 15:19:47 dist-test-slave-c5d6 krb5kdc[29900](info): commencing operation
krb5kdc: starting...
W20260925 15:19:48.579401 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:48 dist-test-slave-c5d6 krb5kdc[29900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:48.697274 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:35895
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:35895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:48.778534 29913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:48.778829 29913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:48.778872 29913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:48.780089 29913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:48.780167 29913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:48.780195 29913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:48.780222 29913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:48.780253 29913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:48.781757 29913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:35895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:35895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:48.782080 29913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:48.782332 29913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:48.787190 29919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:48.787176 29921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:48.787176 29918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:48.787452 29913 server_base.cc:1034] running on GCE node
I20260925 15:19:48.787767 29913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:48.788108 29913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:48.789275 29913 hybrid_clock.cc:648] HybridClock initialized: now 1790349588789250 us; error 36 us; skew 500 ppm
Sep 25 15:19:48 dist-test-slave-c5d6 krb5kdc[29900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349588, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:48.791828 29913 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:48.792323 29913 webserver.cc:460] Webserver started at http://127.8.43.190:46153/ using document root <none> and password file <none>
I20260925 15:19:48.792551 29913 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:48.792618 29913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:48.792768 29913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:48.793818 29913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "7e48debc8ba0430aa7eb07c23afd1826"
format_stamp: "Formatted at 2026-09-25 15:19:48 on dist-test-slave-c5d6"
server_key: "4690b19a4a1574e4124a67027ce8a315"
server_key_iv: "671aa187f7313d89fb11f5abffe9848c"
server_key_version: "encryptionkey@0"
I20260925 15:19:48.794204 29913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "7e48debc8ba0430aa7eb07c23afd1826"
format_stamp: "Formatted at 2026-09-25 15:19:48 on dist-test-slave-c5d6"
server_key: "4690b19a4a1574e4124a67027ce8a315"
server_key_iv: "671aa187f7313d89fb11f5abffe9848c"
server_key_version: "encryptionkey@0"
I20260925 15:19:48.795552 29913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:19:48.796285 29928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:48.796455 29913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:48.796563 29913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "7e48debc8ba0430aa7eb07c23afd1826"
format_stamp: "Formatted at 2026-09-25 15:19:48 on dist-test-slave-c5d6"
server_key: "4690b19a4a1574e4124a67027ce8a315"
server_key_iv: "671aa187f7313d89fb11f5abffe9848c"
server_key_version: "encryptionkey@0"
I20260925 15:19:48.796638 29913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:48.807893 29913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:48.809598 29913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:48.809754 29913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:48.813661 29913 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:35895
I20260925 15:19:48.813696 29980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:35895 every 8 connection(s)
I20260925 15:19:48.814074 29913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:48.814865 29981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:48.817081 29981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826: Bootstrap starting.
I20260925 15:19:48.817574 29981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:48.817799 29981 log.cc:824] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:48.818389 29981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826: No bootstrap required, opened a new log
I20260925 15:19:48.819990 29981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e48debc8ba0430aa7eb07c23afd1826" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35895 } }
I20260925 15:19:48.820119 29981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:48.820142 29981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e48debc8ba0430aa7eb07c23afd1826, State: Initialized, Role: FOLLOWER
I20260925 15:19:48.820233 29981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e48debc8ba0430aa7eb07c23afd1826" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35895 } }
I20260925 15:19:48.820274 29981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:48.820293 29981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:48.820359 29981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:48.821146 29981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e48debc8ba0430aa7eb07c23afd1826" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35895 } }
I20260925 15:19:48.821264 29981 leader_election.cc:302] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e48debc8ba0430aa7eb07c23afd1826; no voters:
I20260925 15:19:48.821429 29981 leader_election.cc:288] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:48.821521 29984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:48.821673 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 29913
I20260925 15:19:48.821702 29981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:48.821719 29984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [term 1 LEADER]: Becoming Leader. State: Replica: 7e48debc8ba0430aa7eb07c23afd1826, State: Running, Role: LEADER
I20260925 15:19:48.821784 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:48.821821 29984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e48debc8ba0430aa7eb07c23afd1826" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35895 } }
I20260925 15:19:48.821972 8366 external_mini_cluster.cc:1428] Setting key 6cba9bb0603f5ece38604d2856c2893f
I20260925 15:19:48.822149 29985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e48debc8ba0430aa7eb07c23afd1826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e48debc8ba0430aa7eb07c23afd1826" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35895 } } }
I20260925 15:19:48.822225 29985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:48.822404 29986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e48debc8ba0430aa7eb07c23afd1826. Latest consensus state: current_term: 1 leader_uuid: "7e48debc8ba0430aa7eb07c23afd1826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e48debc8ba0430aa7eb07c23afd1826" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 35895 } } }
I20260925 15:19:48.822461 29986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:48.822482 29993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:48.822968 29993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:48.824445 29993 catalog_manager.cc:1347] Generated new cluster ID: b55f4e29d0d640268e416982473b9a8b
I20260925 15:19:48.824493 29993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:48.829444 29993 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:48.829908 29993 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 15:19:48 dist-test-slave-c5d6 krb5kdc[29900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:48.834460 29993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e48debc8ba0430aa7eb07c23afd1826: Generated new TSK 0
I20260925 15:19:48.834631 29993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:48.835304 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:48.822572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46615 (local address 127.8.43.190:35895)
0925 15:19:48.823030 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:48.823039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:48.823074 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:19:48.823194 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:48.823201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:48.823583 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:48.824501 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:48.824605 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:48.825989 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:48.827111 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:48.827737 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:48.828181 (+ 444us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:48.830771 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:48.830774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:48.830775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:48.830786 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:48.833211 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:48.833695 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:48.833698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:48.833698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:48.833803 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:48.834070 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:48.834071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:48.834071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:48.834339 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:48.834874 (+ 535us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:48.834971 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 15:19:48.835074 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":67,"threads_started":1}
I20260925 15:19:48.836357 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:48.844265 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:48.837722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47319 (local address 127.8.43.190:35895)
0925 15:19:48.837761 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:48.837763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:48.837814 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:19:48.837879 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:48.837880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:48.837889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:48.837939 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:48.838376 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:48.839207 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:48.839966 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:48.840274 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:48.840343 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:48.841490 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:48.841491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:48.841492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:48.841499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:48.842965 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:48.843408 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:48.843409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:48.843409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:48.843469 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:48.843677 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:48.843678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:48.843678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:48.843801 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:48.843904 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:48.844111 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:19:48.844175 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:19:48.846195 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 29913
W20260925 15:19:48.850033 29898 connection.cc:535] client connection to 127.8.43.190:35895 recv error: Network error: failed to read from TLS socket (remote: 127.8.43.190:35895): Connection reset by peer (error 104)
2026-09-25T15:19:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:48 dist-test-slave-c5d6 krb5kdc[30012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:48 dist-test-slave-c5d6 krb5kdc[30012](info): set up 2 sockets
Sep 25 15:19:48 dist-test-slave-c5d6 krb5kdc[30012](info): commencing operation
krb5kdc: starting...
W20260925 15:19:49.871377 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:49 dist-test-slave-c5d6 krb5kdc[30012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:49.992504 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:42569
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:42569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:50.073714 30025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:50.073978 30025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:50.074018 30025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:50.075151 30025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:50.075219 30025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:50.075260 30025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:50.075291 30025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:50.075335 30025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:50.076817 30025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:42569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:42569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:50.077088 30025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:50.077310 30025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:50.082031 30030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:50.082042 30031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:50.082275 30033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:50.082255 30025 server_base.cc:1034] running on GCE node
I20260925 15:19:50.082538 30025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:50.082831 30025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:50.083974 30025 hybrid_clock.cc:648] HybridClock initialized: now 1790349590083961 us; error 34 us; skew 500 ppm
Sep 25 15:19:50 dist-test-slave-c5d6 krb5kdc[30012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:50.089905 30025 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:50.090390 30025 webserver.cc:460] Webserver started at http://127.8.43.190:34787/ using document root <none> and password file <none>
I20260925 15:19:50.090631 30025 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:50.090677 30025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:50.090840 30025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:50.091970 30025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "be8d87d623c3424d9878edfdbbb6516b"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-c5d6"
server_key: "31d3b24ea27fc01986481e43e0160714"
server_key_iv: "714b633df8c11e548a5a3eba093c2142"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.092322 30025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "be8d87d623c3424d9878edfdbbb6516b"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-c5d6"
server_key: "31d3b24ea27fc01986481e43e0160714"
server_key_iv: "714b633df8c11e548a5a3eba093c2142"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.093681 30025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:19:50.094501 30040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:50.094704 30025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:50.094813 30025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "be8d87d623c3424d9878edfdbbb6516b"
format_stamp: "Formatted at 2026-09-25 15:19:50 on dist-test-slave-c5d6"
server_key: "31d3b24ea27fc01986481e43e0160714"
server_key_iv: "714b633df8c11e548a5a3eba093c2142"
server_key_version: "encryptionkey@0"
I20260925 15:19:50.094888 30025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:50.105094 30025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:50.106945 30025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:50.107100 30025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:50.111166 30025 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:42569
I20260925 15:19:50.111213 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:42569 every 8 connection(s)
I20260925 15:19:50.111601 30025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:50.112500 30093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:50.114880 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b: Bootstrap starting.
I20260925 15:19:50.115538 30093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:50.115842 30093 log.cc:824] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:50.116573 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b: No bootstrap required, opened a new log
I20260925 15:19:50.117835 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30025
I20260925 15:19:50.117980 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:50.118177 8366 external_mini_cluster.cc:1428] Setting key 1bf998648855ea33ac623469ca3c2d3e
I20260925 15:19:50.118881 30093 raft_consensus.cc:348] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d87d623c3424d9878edfdbbb6516b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42569 } }
I20260925 15:19:50.119071 30093 raft_consensus.cc:374] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:50.119122 30093 raft_consensus.cc:729] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8d87d623c3424d9878edfdbbb6516b, State: Initialized, Role: FOLLOWER
I20260925 15:19:50.119288 30093 consensus_queue.cc:260] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d87d623c3424d9878edfdbbb6516b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42569 } }
I20260925 15:19:50.119385 30093 raft_consensus.cc:388] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:50.119424 30093 raft_consensus.cc:482] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:50.119484 30093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:50.120054 30093 raft_consensus.cc:504] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d87d623c3424d9878edfdbbb6516b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42569 } }
I20260925 15:19:50.120144 30093 leader_election.cc:302] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be8d87d623c3424d9878edfdbbb6516b; no voters:
I20260925 15:19:50.120339 30093 leader_election.cc:288] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:50.120366 30098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:50.120492 30098 raft_consensus.cc:686] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [term 1 LEADER]: Becoming Leader. State: Replica: be8d87d623c3424d9878edfdbbb6516b, State: Running, Role: LEADER
I20260925 15:19:50.120615 30093 sys_catalog.cc:564] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:50.120638 30098 consensus_queue.cc:237] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d87d623c3424d9878edfdbbb6516b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42569 } }
I20260925 15:19:50.121034 30098 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [sys.catalog]: SysCatalogTable state changed. Reason: New leader be8d87d623c3424d9878edfdbbb6516b. Latest consensus state: current_term: 1 leader_uuid: "be8d87d623c3424d9878edfdbbb6516b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d87d623c3424d9878edfdbbb6516b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42569 } } }
I20260925 15:19:50.121202 30098 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:50.121050 30099 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be8d87d623c3424d9878edfdbbb6516b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d87d623c3424d9878edfdbbb6516b" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 42569 } } }
I20260925 15:19:50.121402 30099 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:50.121546 30105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:50.122145 30105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:50.123871 30105 catalog_manager.cc:1347] Generated new cluster ID: 5b53fb13e2b04c0d8da8dc41af5c3408
I20260925 15:19:50.123929 30105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:50.124394 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:50.118740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46529 (local address 127.8.43.190:42569)
0925 15:19:50.118913 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:50.118921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:50.118961 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:19:50.119060 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:50.119066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:50.119132 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:50.119320 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:50.119552 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.121030 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.122174 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.122706 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.122837 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:50.123625 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:50.123629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:50.123629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:50.123843 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:50.123880 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:50.123895 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:50.123992 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:50.124174 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 15:19:50.124199 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260925 15:19:50.125389 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:50.130254 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:50.126802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39095 (local address 127.8.43.190:42569)
0925 15:19:50.126831 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:50.126833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:50.126965 (+ 132us) server_negotiation.cc:407] Connection header received
0925 15:19:50.127047 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:50.127048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:50.127066 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:50.127092 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:50.127275 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.127942 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.128927 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:50.129247 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:50.129281 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:50.130026 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:50.130027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:50.130028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:50.130031 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:50.130043 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:50.130046 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:50.130074 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:50.130178 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:19:50.130189 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 15:19:50.147822 30105 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:50.148626 30105 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:50.154413 30105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be8d87d623c3424d9878edfdbbb6516b: Generated new TSK 0
I20260925 15:19:50.154598 30105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:50.157614 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30025
W20260925 15:19:50.161710 30010 connection.cc:535] client connection to 127.8.43.190:42569 recv error: Network error: failed to read from TLS socket (remote: 127.8.43.190:42569): Connection reset by peer (error 104)
2026-09-25T15:19:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:50 dist-test-slave-c5d6 krb5kdc[30124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:50 dist-test-slave-c5d6 krb5kdc[30124](info): set up 2 sockets
Sep 25 15:19:50 dist-test-slave-c5d6 krb5kdc[30124](info): commencing operation
krb5kdc: starting...
W20260925 15:19:51.187301 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:51 dist-test-slave-c5d6 krb5kdc[30124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:51.302248 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38325
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:38325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:51.382288 30137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:51.382608 30137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:51.382712 30137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:51.383975 30137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:51.384053 30137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:51.384078 30137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:51.384109 30137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:51.384142 30137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:51.385636 30137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:38325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:51.385926 30137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:51.386176 30137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:51.391088 30143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:51.391099 30145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:51.391130 30142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:51.391211 30137 server_base.cc:1034] running on GCE node
I20260925 15:19:51.391585 30137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:51.391844 30137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:51.392999 30137 hybrid_clock.cc:648] HybridClock initialized: now 1790349591392976 us; error 37 us; skew 500 ppm
Sep 25 15:19:51 dist-test-slave-c5d6 krb5kdc[30124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:51.395512 30137 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:51.396011 30137 webserver.cc:460] Webserver started at http://127.8.43.190:34509/ using document root <none> and password file <none>
I20260925 15:19:51.396235 30137 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:51.396302 30137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:51.396452 30137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:51.397553 30137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "10dc6d349ee84d99b823e11aa921236a"
format_stamp: "Formatted at 2026-09-25 15:19:51 on dist-test-slave-c5d6"
server_key: "28590edf2cfa0880a4284f28db391b2a"
server_key_iv: "c916bd59b99d02305c68c50b293bf569"
server_key_version: "encryptionkey@0"
I20260925 15:19:51.397917 30137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "10dc6d349ee84d99b823e11aa921236a"
format_stamp: "Formatted at 2026-09-25 15:19:51 on dist-test-slave-c5d6"
server_key: "28590edf2cfa0880a4284f28db391b2a"
server_key_iv: "c916bd59b99d02305c68c50b293bf569"
server_key_version: "encryptionkey@0"
I20260925 15:19:51.399314 30137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:19:51.400060 30152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:51.400238 30137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:51.400338 30137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "10dc6d349ee84d99b823e11aa921236a"
format_stamp: "Formatted at 2026-09-25 15:19:51 on dist-test-slave-c5d6"
server_key: "28590edf2cfa0880a4284f28db391b2a"
server_key_iv: "c916bd59b99d02305c68c50b293bf569"
server_key_version: "encryptionkey@0"
I20260925 15:19:51.400425 30137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:51.403770 30137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:51.406045 30137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:51.406200 30137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:51.410101 30137 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:38325
I20260925 15:19:51.410131 30204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:38325 every 8 connection(s)
I20260925 15:19:51.410462 30137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:51.411481 30205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:51.413710 30205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a: Bootstrap starting.
I20260925 15:19:51.414285 30205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:51.414562 30205 log.cc:824] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:51.415288 30205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a: No bootstrap required, opened a new log
I20260925 15:19:51.416477 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30137
I20260925 15:19:51.416563 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:51.416741 8366 external_mini_cluster.cc:1428] Setting key 027324f506d022aa8e026502f1133100
I20260925 15:19:51.417002 30205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dc6d349ee84d99b823e11aa921236a" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38325 } }
I20260925 15:19:51.417182 30205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:51.417224 30205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dc6d349ee84d99b823e11aa921236a, State: Initialized, Role: FOLLOWER
I20260925 15:19:51.417343 30205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dc6d349ee84d99b823e11aa921236a" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38325 } }
I20260925 15:19:51.417433 30205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:51.417469 30205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:51.417567 30205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:51.418730 30205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dc6d349ee84d99b823e11aa921236a" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38325 } }
I20260925 15:19:51.418839 30205 leader_election.cc:302] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10dc6d349ee84d99b823e11aa921236a; no voters:
I20260925 15:19:51.419013 30205 leader_election.cc:288] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:51.419042 30210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:51.419025 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:51.417264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39541 (local address 127.8.43.190:38325)
0925 15:19:51.417444 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:51.417454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:51.417491 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:19:51.417630 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:51.417637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:51.417680 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:51.417790 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:51.418430 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:51.418433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:51.418433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:51.418437 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:51.418479 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:51.418490 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:51.418578 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:51.418687 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:19:51.418749 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260925 15:19:51.419160 30210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [term 1 LEADER]: Becoming Leader. State: Replica: 10dc6d349ee84d99b823e11aa921236a, State: Running, Role: LEADER
I20260925 15:19:51.419283 30205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:51.419308 30210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dc6d349ee84d99b823e11aa921236a" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38325 } }
I20260925 15:19:51.419664 30212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10dc6d349ee84d99b823e11aa921236a. Latest consensus state: current_term: 1 leader_uuid: "10dc6d349ee84d99b823e11aa921236a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dc6d349ee84d99b823e11aa921236a" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38325 } } }
I20260925 15:19:51.419708 30211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10dc6d349ee84d99b823e11aa921236a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10dc6d349ee84d99b823e11aa921236a" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38325 } } }
I20260925 15:19:51.419785 30212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:51.419852 30211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:51.420073 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:51.420210 30219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:51.420739 30219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:51.422621 30219 catalog_manager.cc:1347] Generated new cluster ID: b4b2e680ada6453f9b31b61153374420
I20260925 15:19:51.422677 30219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:51.427309 30219 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:51.427455 30219 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:51.428105 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:51.421680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41491 (local address 127.8.43.190:38325)
0925 15:19:51.421713 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:51.421714 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:51.427092 (+ 5378us) server_negotiation.cc:407] Connection header received
0925 15:19:51.427199 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:51.427200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:51.427206 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:51.427286 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:51.427892 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:51.427893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:51.427893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:51.427897 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:51.427912 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:51.427916 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:51.427975 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:51.428035 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:19:51.428047 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:19:51.435411 30219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10dc6d349ee84d99b823e11aa921236a: Generated new TSK 0
I20260925 15:19:51.435607 30219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:51.441236 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30137
2026-09-25T15:19:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:51 dist-test-slave-c5d6 krb5kdc[30236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:51 dist-test-slave-c5d6 krb5kdc[30236](info): set up 2 sockets
Sep 25 15:19:51 dist-test-slave-c5d6 krb5kdc[30236](info): commencing operation
krb5kdc: starting...
W20260925 15:19:52.479389 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:52 dist-test-slave-c5d6 krb5kdc[30236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:52.594198 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:41741
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:41741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:52.674258 30249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:52.674467 30249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:52.674490 30249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:52.675679 30249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:52.675721 30249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:52.675735 30249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:52.675748 30249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:52.675798 30249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:52.677219 30249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:41741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:52.677459 30249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:52.677699 30249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:52.682433 30254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:52.682467 30255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:52.682628 30249 server_base.cc:1034] running on GCE node
W20260925 15:19:52.682437 30257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:52.682921 30249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:52.683171 30249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:52.684309 30249 hybrid_clock.cc:648] HybridClock initialized: now 1790349592684283 us; error 43 us; skew 500 ppm
Sep 25 15:19:52 dist-test-slave-c5d6 krb5kdc[30236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:52.686676 30249 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:52.687095 30249 webserver.cc:460] Webserver started at http://127.8.43.190:33729/ using document root <none> and password file <none>
I20260925 15:19:52.687342 30249 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:52.687390 30249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:52.687489 30249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:52.688480 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "4c0d7a214d1b4e879380bb1d35123284"
format_stamp: "Formatted at 2026-09-25 15:19:52 on dist-test-slave-c5d6"
server_key: "a692b4f06a5f1fd633614100454b1178"
server_key_iv: "b129dba672002c8f8ced13440b34e9ee"
server_key_version: "encryptionkey@0"
I20260925 15:19:52.688776 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "4c0d7a214d1b4e879380bb1d35123284"
format_stamp: "Formatted at 2026-09-25 15:19:52 on dist-test-slave-c5d6"
server_key: "a692b4f06a5f1fd633614100454b1178"
server_key_iv: "b129dba672002c8f8ced13440b34e9ee"
server_key_version: "encryptionkey@0"
I20260925 15:19:52.690043 30249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:19:52.690716 30264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:52.690881 30249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:19:52.690963 30249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "4c0d7a214d1b4e879380bb1d35123284"
format_stamp: "Formatted at 2026-09-25 15:19:52 on dist-test-slave-c5d6"
server_key: "a692b4f06a5f1fd633614100454b1178"
server_key_iv: "b129dba672002c8f8ced13440b34e9ee"
server_key_version: "encryptionkey@0"
I20260925 15:19:52.691015 30249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:52.709653 30249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:52.711552 30249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:52.711659 30249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:52.715639 30249 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:41741
I20260925 15:19:52.715687 30316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:41741 every 8 connection(s)
I20260925 15:19:52.716025 30249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:52.716879 30317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:52.718575 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30249
I20260925 15:19:52.718707 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:52.718890 8366 external_mini_cluster.cc:1428] Setting key 8cb89eda407535fc194b6b2a6f613b52
I20260925 15:19:52.719434 30317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284: Bootstrap starting.
I20260925 15:19:52.720021 30317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:52.720299 30317 log.cc:824] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:52.721006 30317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284: No bootstrap required, opened a new log
I20260925 15:19:52.722690 30317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0d7a214d1b4e879380bb1d35123284" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41741 } }
I20260925 15:19:52.722841 30317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:52.722887 30317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0d7a214d1b4e879380bb1d35123284, State: Initialized, Role: FOLLOWER
I20260925 15:19:52.723040 30317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0d7a214d1b4e879380bb1d35123284" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41741 } }
I20260925 15:19:52.723131 30317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:52.723176 30317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:52.723242 30317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:52.723778 30317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0d7a214d1b4e879380bb1d35123284" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41741 } }
I20260925 15:19:52.723888 30317 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0d7a214d1b4e879380bb1d35123284; no voters:
I20260925 15:19:52.724088 30317 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:52.724107 30322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:52.724220 30322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [term 1 LEADER]: Becoming Leader. State: Replica: 4c0d7a214d1b4e879380bb1d35123284, State: Running, Role: LEADER
I20260925 15:19:52.724342 30317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:52.724351 30322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0d7a214d1b4e879380bb1d35123284" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41741 } }
I20260925 15:19:52.724901 30323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c0d7a214d1b4e879380bb1d35123284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0d7a214d1b4e879380bb1d35123284" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41741 } } }
I20260925 15:19:52.724907 30324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0d7a214d1b4e879380bb1d35123284. Latest consensus state: current_term: 1 leader_uuid: "4c0d7a214d1b4e879380bb1d35123284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0d7a214d1b4e879380bb1d35123284" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 41741 } } }
I20260925 15:19:52.724998 30323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:52.725025 30324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:52.725196 30329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 15:19:52 dist-test-slave-c5d6 krb5kdc[30236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:52.725787 30329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:52.727936 30329 catalog_manager.cc:1347] Generated new cluster ID: 855a2950a1f04b2aab0e04539d2903e3
I20260925 15:19:52.728001 30329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:52.729652 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:52.719389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50669 (local address 127.8.43.190:41741)
0925 15:19:52.719580 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:52.719590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:52.719620 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:19:52.720034 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:52.720040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:52.720119 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:52.720272 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:52.721032 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:52.721987 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:52.722746 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:52.723337 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:52.723461 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:52.726197 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:52.726201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:52.726202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:52.726213 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:52.727908 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:52.728279 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:52.728282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:52.728282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:52.728369 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:52.728547 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:52.728548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:52.728548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:52.728804 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:52.728907 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:52.729322 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 15:19:52.729442 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260925 15:19:52.730645 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:52.736515 30329 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:52.737205 30329 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:52.739480 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:52.732286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54305 (local address 127.8.43.190:41741)
0925 15:19:52.732319 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:52.732321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:52.732377 (+ 56us) server_negotiation.cc:407] Connection header received
0925 15:19:52.732447 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:52.732448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:52.732455 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:52.732503 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:52.733114 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:52.733829 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:52.734850 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:52.735157 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:52.735236 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:52.736769 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:52.736770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:52.736770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:52.736776 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:52.738314 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:52.738713 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:52.738714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:52.738714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:52.738766 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:52.738962 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:52.738962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:52.738963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:52.739070 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:52.739150 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:52.739313 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:19:52.739390 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:19:52.743445 30329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0d7a214d1b4e879380bb1d35123284: Generated new TSK 0
I20260925 15:19:52.743614 30329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:52.747784 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30249
2026-09-25T15:19:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:52 dist-test-slave-c5d6 krb5kdc[30348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:52 dist-test-slave-c5d6 krb5kdc[30348](info): set up 2 sockets
Sep 25 15:19:52 dist-test-slave-c5d6 krb5kdc[30348](info): commencing operation
krb5kdc: starting...
W20260925 15:19:53.779462 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:53 dist-test-slave-c5d6 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:53.898419 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44815
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:53.977656 30361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:53.977892 30361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:53.977939 30361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:53.979092 30361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:53.979162 30361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:53.979187 30361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:53.979212 30361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:53.979236 30361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:53.980731 30361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:44815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:53.981005 30361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:53.981227 30361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:53.985992 30366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:53.986002 30369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:53.986188 30367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:53.986382 30361 server_base.cc:1034] running on GCE node
I20260925 15:19:53.986590 30361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:53.986894 30361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:53.988070 30361 hybrid_clock.cc:648] HybridClock initialized: now 1790349593988058 us; error 32 us; skew 500 ppm
Sep 25 15:19:53 dist-test-slave-c5d6 krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349593, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:53.990514 30361 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:53.990978 30361 webserver.cc:460] Webserver started at http://127.8.43.190:35475/ using document root <none> and password file <none>
I20260925 15:19:53.991209 30361 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:53.991329 30361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:53.991472 30361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:53.992498 30361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "be038d8520bc4c8590f8d2f2b6ddafd7"
format_stamp: "Formatted at 2026-09-25 15:19:53 on dist-test-slave-c5d6"
server_key: "dd38c9be714bb2808bd94866e6b627dd"
server_key_iv: "ab9df312eb9a383482cce6440f765353"
server_key_version: "encryptionkey@0"
I20260925 15:19:53.992852 30361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "be038d8520bc4c8590f8d2f2b6ddafd7"
format_stamp: "Formatted at 2026-09-25 15:19:53 on dist-test-slave-c5d6"
server_key: "dd38c9be714bb2808bd94866e6b627dd"
server_key_iv: "ab9df312eb9a383482cce6440f765353"
server_key_version: "encryptionkey@0"
I20260925 15:19:53.994215 30361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:19:53.995121 30376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:53.995362 30361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:53.995476 30361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "be038d8520bc4c8590f8d2f2b6ddafd7"
format_stamp: "Formatted at 2026-09-25 15:19:53 on dist-test-slave-c5d6"
server_key: "dd38c9be714bb2808bd94866e6b627dd"
server_key_iv: "ab9df312eb9a383482cce6440f765353"
server_key_version: "encryptionkey@0"
I20260925 15:19:53.995568 30361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:54.036175 30361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:54.038162 30361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:54.038326 30361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:54.042393 30361 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:44815
I20260925 15:19:54.042431 30428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:44815 every 8 connection(s)
I20260925 15:19:54.042809 30361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:54.043413 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30361
I20260925 15:19:54.043520 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:54.043721 8366 external_mini_cluster.cc:1428] Setting key f712e3945b6198aaa1f3624ccc9c0df7
I20260925 15:19:54.043893 30429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:54.046753 30429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7: Bootstrap starting.
I20260925 15:19:54.047412 30429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:54.047710 30429 log.cc:824] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:54.048456 30429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7: No bootstrap required, opened a new log
I20260925 15:19:54.049059 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:54.044250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36137 (local address 127.8.43.190:44815)
0925 15:19:54.044462 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:54.044470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:54.044494 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:19:54.044986 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:54.044992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:54.045037 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:54.045183 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:54.045383 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.046265 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.047132 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.047661 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.047779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.048435 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:54.048439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:54.048440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:54.048639 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:54.048676 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:54.048686 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:54.048750 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:54.048821 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:19:54.048881 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260925 15:19:54.050107 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:54.050666 30429 raft_consensus.cc:348] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be038d8520bc4c8590f8d2f2b6ddafd7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44815 } }
I20260925 15:19:54.050819 30429 raft_consensus.cc:374] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:54.050864 30429 raft_consensus.cc:729] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be038d8520bc4c8590f8d2f2b6ddafd7, State: Initialized, Role: FOLLOWER
I20260925 15:19:54.051013 30429 consensus_queue.cc:260] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be038d8520bc4c8590f8d2f2b6ddafd7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44815 } }
I20260925 15:19:54.051105 30429 raft_consensus.cc:388] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:54.051148 30429 raft_consensus.cc:482] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:54.051218 30429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:54.051834 30429 raft_consensus.cc:504] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be038d8520bc4c8590f8d2f2b6ddafd7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44815 } }
I20260925 15:19:54.051954 30429 leader_election.cc:302] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be038d8520bc4c8590f8d2f2b6ddafd7; no voters:
I20260925 15:19:54.052138 30429 leader_election.cc:288] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:54.052168 30439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:54.052270 30439 raft_consensus.cc:686] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [term 1 LEADER]: Becoming Leader. State: Replica: be038d8520bc4c8590f8d2f2b6ddafd7, State: Running, Role: LEADER
I20260925 15:19:54.052380 30429 sys_catalog.cc:564] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:54.052402 30439 consensus_queue.cc:237] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be038d8520bc4c8590f8d2f2b6ddafd7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44815 } }
I20260925 15:19:54.052770 30440 sys_catalog.cc:455] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be038d8520bc4c8590f8d2f2b6ddafd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be038d8520bc4c8590f8d2f2b6ddafd7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44815 } } }
I20260925 15:19:54.052914 30440 sys_catalog.cc:458] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:54.053121 30440 sys_catalog.cc:455] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be038d8520bc4c8590f8d2f2b6ddafd7. Latest consensus state: current_term: 1 leader_uuid: "be038d8520bc4c8590f8d2f2b6ddafd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be038d8520bc4c8590f8d2f2b6ddafd7" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 44815 } } }
I20260925 15:19:54.053224 30440 sys_catalog.cc:458] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:54.053440 30448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:54.054059 30448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:54.054864 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:54.051510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49429 (local address 127.8.43.190:44815)
0925 15:19:54.051546 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:54.051548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:54.051594 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:19:54.051659 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:54.051659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:54.051669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:54.051718 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:54.051831 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.052453 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.053416 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:54.053727 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:54.053821 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:54.054502 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:54.054503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:54.054503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:54.054507 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:54.054519 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:54.054523 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:54.054562 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:54.054753 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 15:19:54.054795 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:19:54.055553 30448 catalog_manager.cc:1347] Generated new cluster ID: b3cc17ed7ab04c1f8dca957cfb01b1d6
I20260925 15:19:54.055603 30448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:54.062300 30448 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:54.062994 30448 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:54.067551 30448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be038d8520bc4c8590f8d2f2b6ddafd7: Generated new TSK 0
I20260925 15:19:54.067706 30448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:54.070066 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30361
2026-09-25T15:19:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:54 dist-test-slave-c5d6 krb5kdc[30461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:54 dist-test-slave-c5d6 krb5kdc[30461](info): set up 2 sockets
Sep 25 15:19:54 dist-test-slave-c5d6 krb5kdc[30461](info): commencing operation
krb5kdc: starting...
W20260925 15:19:55.123420 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:55 dist-test-slave-c5d6 krb5kdc[30461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:55.241911 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:36085
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:36085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:55.321914 30474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:55.322150 30474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:55.322186 30474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:55.323344 30474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:55.323415 30474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:55.323437 30474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:55.323472 30474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:55.323503 30474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:55.324918 30474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:36085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:55.325189 30474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:55.325408 30474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:55.330211 30479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:55.330250 30482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:55.330408 30474 server_base.cc:1034] running on GCE node
W20260925 15:19:55.330230 30480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:55.330742 30474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:55.331027 30474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:55.332195 30474 hybrid_clock.cc:648] HybridClock initialized: now 1790349595332172 us; error 55 us; skew 500 ppm
Sep 25 15:19:55 dist-test-slave-c5d6 krb5kdc[30461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349595, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:55.338950 30474 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:55.339499 30474 webserver.cc:460] Webserver started at http://127.8.43.190:40381/ using document root <none> and password file <none>
I20260925 15:19:55.339727 30474 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:55.339792 30474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:55.339947 30474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:55.340996 30474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "b7854f93bde640cdadf8be2d2d9927a8"
format_stamp: "Formatted at 2026-09-25 15:19:55 on dist-test-slave-c5d6"
server_key: "0d4d3ccc1f00716e6234e762852af82a"
server_key_iv: "50979bacffc52072bcb4546ec458959f"
server_key_version: "encryptionkey@0"
I20260925 15:19:55.341359 30474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "b7854f93bde640cdadf8be2d2d9927a8"
format_stamp: "Formatted at 2026-09-25 15:19:55 on dist-test-slave-c5d6"
server_key: "0d4d3ccc1f00716e6234e762852af82a"
server_key_iv: "50979bacffc52072bcb4546ec458959f"
server_key_version: "encryptionkey@0"
I20260925 15:19:55.342650 30474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:55.343402 30489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:55.343580 30474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:55.343677 30474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "b7854f93bde640cdadf8be2d2d9927a8"
format_stamp: "Formatted at 2026-09-25 15:19:55 on dist-test-slave-c5d6"
server_key: "0d4d3ccc1f00716e6234e762852af82a"
server_key_iv: "50979bacffc52072bcb4546ec458959f"
server_key_version: "encryptionkey@0"
I20260925 15:19:55.343763 30474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:55.346995 30474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:55.349222 30474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:55.349375 30474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:55.353129 30474 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:36085
I20260925 15:19:55.353158 30541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:36085 every 8 connection(s)
I20260925 15:19:55.353473 30474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:55.354387 30542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:55.356364 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30474
I20260925 15:19:55.356495 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:55.356675 8366 external_mini_cluster.cc:1428] Setting key 276716e6352a5b44481ecd48af00d200
I20260925 15:19:55.356689 30542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8: Bootstrap starting.
I20260925 15:19:55.357360 30542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:55.357666 30542 log.cc:824] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:55.358502 30542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8: No bootstrap required, opened a new log
I20260925 15:19:55.359150 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:55.357171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46813 (local address 127.8.43.190:36085)
0925 15:19:55.357350 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:55.357356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:55.357380 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:19:55.357879 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:55.357886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:55.357918 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:55.358076 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:55.358756 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:55.358759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:55.358759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:55.358764 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:55.358812 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:55.358822 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:55.358915 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:55.358955 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 15:19:55.358993 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260925 15:19:55.360081 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:55.360956 30542 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7854f93bde640cdadf8be2d2d9927a8" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36085 } }
I20260925 15:19:55.361135 30542 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:19:55.361184 30542 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7854f93bde640cdadf8be2d2d9927a8, State: Initialized, Role: FOLLOWER
I20260925 15:19:55.361308 30542 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7854f93bde640cdadf8be2d2d9927a8" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36085 } }
I20260925 15:19:55.361395 30542 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:19:55.361438 30542 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:19:55.361514 30542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:55.362131 30542 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7854f93bde640cdadf8be2d2d9927a8" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36085 } }
I20260925 15:19:55.362252 30542 leader_election.cc:302] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7854f93bde640cdadf8be2d2d9927a8; no voters:
I20260925 15:19:55.362416 30542 leader_election.cc:288] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:19:55.362447 30552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:55.362565 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:55.361513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56201 (local address 127.8.43.190:36085)
0925 15:19:55.361557 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:55.361559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:55.361695 (+ 136us) server_negotiation.cc:407] Connection header received
0925 15:19:55.361719 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:55.361720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:55.361724 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:55.361787 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:55.362292 (+ 505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:55.362293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:55.362293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:19:55.362298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:55.362312 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:19:55.362316 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:55.362378 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:55.362425 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:19:55.362438 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:19:55.362661 30552 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [term 1 LEADER]: Becoming Leader. State: Replica: b7854f93bde640cdadf8be2d2d9927a8, State: Running, Role: LEADER
I20260925 15:19:55.362710 30542 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:55.362740 30552 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7854f93bde640cdadf8be2d2d9927a8" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36085 } }
I20260925 15:19:55.363030 30552 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7854f93bde640cdadf8be2d2d9927a8. Latest consensus state: current_term: 1 leader_uuid: "b7854f93bde640cdadf8be2d2d9927a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7854f93bde640cdadf8be2d2d9927a8" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36085 } } }
I20260925 15:19:55.363040 30553 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7854f93bde640cdadf8be2d2d9927a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7854f93bde640cdadf8be2d2d9927a8" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 36085 } } }
I20260925 15:19:55.363118 30552 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:55.363126 30553 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:55.363390 30559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:55.364058 30559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:55.365545 30559 catalog_manager.cc:1347] Generated new cluster ID: 3e0d184998ef4d9cb1fa3793694dd5e6
I20260925 15:19:55.365592 30559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:55.395681 30559 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:55.395891 30559 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:55.401760 30559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7854f93bde640cdadf8be2d2d9927a8: Generated new TSK 0
I20260925 15:19:55.401968 30559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:55.406247 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30474
2026-09-25T15:19:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1336 ms)
[----------] 18 tests from AuthTokenIssuingTest (23618 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:55 dist-test-slave-c5d6 krb5kdc[30573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:55 dist-test-slave-c5d6 krb5kdc[30573](info): set up 2 sockets
Sep 25 15:19:55 dist-test-slave-c5d6 krb5kdc[30573](info): commencing operation
krb5kdc: starting...
W20260925 15:19:56.439795 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:56.567606 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:40255
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:40255,127.8.43.189:36193,127.8.43.188:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:56.649894 30586 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:19:56.650125 30586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:56.650144 30586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:56.650166 30586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:56.651428 30586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:56.651474 30586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:56.651491 30586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:56.651505 30586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:56.651551 30586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:56.653050 30586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:40255,127.8.43.189:36193,127.8.43.188:39269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:56.653311 30586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:56.653555 30586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:56.658414 30594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:56.658414 30592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:56.658663 30586 server_base.cc:1034] running on GCE node
W20260925 15:19:56.658447 30591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:56.659006 30586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:56.659327 30586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:56.660483 30586 hybrid_clock.cc:648] HybridClock initialized: now 1790349596660458 us; error 39 us; skew 500 ppm
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:56.663095 30586 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:56.663585 30586 webserver.cc:460] Webserver started at http://127.8.43.190:43811/ using document root <none> and password file <none>
I20260925 15:19:56.663867 30586 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:56.663919 30586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:56.664053 30586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:56.665145 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "3cd067b8da6541eca56b9ea57bafd38d"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "baf30b284899c50c1aa5220227379106"
server_key_iv: "8c285dc0a6d2f50c0022c52092cecb57"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.665504 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "3cd067b8da6541eca56b9ea57bafd38d"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "baf30b284899c50c1aa5220227379106"
server_key_iv: "8c285dc0a6d2f50c0022c52092cecb57"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.666829 30586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:19:56.667675 30601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:56.667881 30586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:56.668015 30586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "3cd067b8da6541eca56b9ea57bafd38d"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "baf30b284899c50c1aa5220227379106"
server_key_iv: "8c285dc0a6d2f50c0022c52092cecb57"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.668093 30586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:56.683867 30586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:56.686211 30586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:56.686323 30586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:56.690249 30586 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:40255
I20260925 15:19:56.690300 30653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:40255 every 8 connection(s)
I20260925 15:19:56.690675 30586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:56.691787 30654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:56.692034 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30586
I20260925 15:19:56.692152 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:56.692351 8366 external_mini_cluster.cc:1428] Setting key 90d9210262b3ef26308f08280d1dbb2c
I20260925 15:19:56.693549 30654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.43.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.189@KRBTEST.COM" created.
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:56.706089 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.694308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:40255 (local address 127.0.0.1:48350)
0925 15:19:56.694478 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.694525 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.695048 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.695336 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.695339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.695614 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.695759 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.695763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.696775 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.696776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.697467 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.697469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.698006 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.698007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.698071 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.698607 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.698615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.702504 (+ 3889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:56.704600 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.704606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.704608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.704801 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.705062 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.705063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.705063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.705148 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.705503 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:56.705508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:56.705568 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 15:19:56.705675 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:19:56.705920 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":91,"thread_start_us":94,"threads_started":1}
I20260925 15:19:56.706089 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.694503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.8.43.190:40255)
0925 15:19:56.694666 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.694670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.694700 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:19:56.695202 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.695204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.695245 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.695322 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.695855 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.696672 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.697566 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.697885 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.698022 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.702606 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:56.702614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:56.702614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:56.702623 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:56.704512 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.704911 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.704914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.704915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.704994 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.705213 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.705214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.705214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.705424 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:56.705510 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:56.705708 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 15:19:56.705840 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260925 15:19:56.706888 30654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:56.707284 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.707112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35514)
0925 15:19:56.707164 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.707254 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:19:56.707442 30605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
I20260925 15:19:56.707850 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.707725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35526)
0925 15:19:56.707790 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.707822 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:19:56.708024 30654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:36193: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab.
Entry for principal kudu/127.8.43.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab.
I20260925 15:19:56.716185 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.189:36193
--webserver_interface=127.8.43.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab
--principal=kudu/127.8.43.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:40255,127.8.43.189:36193,127.8.43.188:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:56.757325 30654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } attempt: 1
I20260925 15:19:56.757915 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.757731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35542)
0925 15:19:56.757806 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.757861 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:19:56.758368 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.758267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35552)
0925 15:19:56.758314 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.758343 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:19:56.758468 30654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:36193: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
W20260925 15:19:56.797086 30661 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:19:56.797299 30661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:56.797317 30661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:56.797336 30661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:56.798400 30661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:56.798435 30661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:56.798449 30661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:56.798461 30661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:56.798473 30661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:56.799985 30661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:40255,127.8.43.189:36193,127.8.43.188:39269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab
--principal=kudu/127.8.43.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.189:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.43.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:56.800231 30661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:56.800416 30661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:56.805143 30666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:56.805119 30669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:56.805135 30667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:56.805330 30661 server_base.cc:1034] running on GCE node
I20260925 15:19:56.805622 30661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:56.805915 30661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:56.807080 30661 hybrid_clock.cc:648] HybridClock initialized: now 1790349596807073 us; error 40 us; skew 500 ppm
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:56.809474 30661 init.cc:377] Logged in from keytab as kudu/127.8.43.189@KRBTEST.COM (short username kudu)
I20260925 15:19:56.809881 30661 webserver.cc:460] Webserver started at http://127.8.43.189:39995/ using document root <none> and password file <none>
I20260925 15:19:56.810109 30661 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:56.810174 30661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:56.810325 30661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:56.811406 30661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/data/instance:
uuid: "6b465c19fed146de9457b7d67ab3f11a"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "a60db32e7c4e00628e60ce84a80ae9b8"
server_key_iv: "113ed6df74989d3ad11d5c065d5290a8"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.811754 30661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/wal/instance:
uuid: "6b465c19fed146de9457b7d67ab3f11a"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "a60db32e7c4e00628e60ce84a80ae9b8"
server_key_iv: "113ed6df74989d3ad11d5c065d5290a8"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.813031 30661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:19:56.813786 30676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:56.813979 30661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:19:56.814086 30661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/wal
uuid: "6b465c19fed146de9457b7d67ab3f11a"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "a60db32e7c4e00628e60ce84a80ae9b8"
server_key_iv: "113ed6df74989d3ad11d5c065d5290a8"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.814165 30661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:56.821980 30661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:56.823886 30661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:56.824044 30661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:56.826606 30654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } attempt: 2
I20260925 15:19:56.827026 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.826887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35566)
0925 15:19:56.826971 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.827000 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:19:56.827440 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.827332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35578)
0925 15:19:56.827390 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.827419 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:19:56.827538 30654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:36193: Network error: Client connection negotiation failed: client connection to 127.8.43.189:36193: connect: Connection refused (error 111)
I20260925 15:19:56.828127 30661 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.189:36193
I20260925 15:19:56.828173 30728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.189:36193 every 8 connection(s)
I20260925 15:19:56.828495 30661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
I20260925 15:19:56.829376 30729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:56.830629 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30661
I20260925 15:19:56.830713 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-1/wal/instance
I20260925 15:19:56.830910 8366 external_mini_cluster.cc:1428] Setting key 8c27990456642a48a44ae4ae8220c392
I20260925 15:19:56.830919 30729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.43.188@KRBTEST.COM; defaulting to no policy
I20260925 15:19:56.840835 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.831610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.8.43.190:40255)
0925 15:19:56.831671 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.831673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.831891 (+ 218us) server_negotiation.cc:407] Connection header received
0925 15:19:56.832572 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.832573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.832587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.832658 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.833197 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.833775 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.835311 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.835505 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.835558 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.837900 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:56.837902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:56.837902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:56.837909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:56.839360 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.839822 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.839823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.839824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.839882 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.840137 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.840138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.840138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.840255 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:56.840330 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:56.840628 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:19:56.840739 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:19:56.840973 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.831597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:40255 (local address 127.0.0.1:48354)
0925 15:19:56.831760 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.831805 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.832475 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.832707 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.832710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.832935 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.833114 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.833118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.833868 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.833869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.835206 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.835209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.835601 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.835603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.835767 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.836251 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.836262 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.837804 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:56.839454 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.839461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.839463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.839736 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.839957 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.839958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.839958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.840068 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.840323 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:56.840329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:56.840467 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:19:56.840575 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:19:56.840735 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.188@KRBTEST.COM" created.
I20260925 15:19:56.841465 30729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:19:56.848906 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.841725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35588)
0925 15:19:56.841765 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.841774 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.841806 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.842061 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.842062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.842192 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.842233 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.842235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.843130 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.843131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.843677 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.843678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.844149 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.844150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.844209 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.844645 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.844648 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.845785 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:56.847959 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.847960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.847960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.848114 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.848341 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.848341 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.848342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.848389 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.848701 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:56.848701 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:56.848733 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:19:56.848780 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:19:56.848850 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 15:19:56.849044 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.841793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.8.43.189:36193)
0925 15:19:56.841937 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.841940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.841957 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:19:56.841972 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.841974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.841983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.842061 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.842312 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.843036 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.843764 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.844065 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.844149 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.845870 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:56.845879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:56.845880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:56.845887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:56.847871 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.848201 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.848205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.848205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.848268 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.848461 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.848462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.848462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.848633 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:56.848722 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:56.848815 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:19:56.848927 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260925 15:19:56.849753 30729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:56.850069 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.849969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:34924)
0925 15:19:56.849998 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.850034 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 15:19:56.850219 30678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
I20260925 15:19:56.850639 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.850553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:34926)
0925 15:19:56.850599 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.850623 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:19:56.850811 30729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:39269: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab.
Entry for principal kudu/127.8.43.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab.
I20260925 15:19:56.857380 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.188:39269
--webserver_interface=127.8.43.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab
--principal=kudu/127.8.43.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:39221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:40255,127.8.43.189:36193,127.8.43.188:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:56.900031 30729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } attempt: 1
I20260925 15:19:56.900501 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.900365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:34942)
0925 15:19:56.900425 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.900470 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:19:56.900929 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.900785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:34948)
0925 15:19:56.900849 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.900892 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:19:56.901085 30729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:39269: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
I20260925 15:19:56.918669 30654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } attempt: 3
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:19:56.925992 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.919020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35596)
0925 15:19:56.919077 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.919087 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.919144 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.919346 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.919347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.919521 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.919584 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.919586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.920271 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.920272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.920816 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.920818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.921526 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.921527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.921585 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.922107 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.922112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.923361 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:56.925105 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.925106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.925106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.925258 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.925465 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.925466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.925466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.925515 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.925774 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:56.925775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:56.925811 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:19:56.925860 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:19:56.925929 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:19:56.926059 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.919084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.8.43.189:36193)
0925 15:19:56.919148 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.919150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.919180 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:19:56.919213 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.919249 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.919261 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.919325 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.919667 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.920183 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.920936 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.921378 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.921518 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.923447 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:56.923449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:56.923449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:56.923457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:56.925026 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.925328 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.925329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.925329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.925392 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.925576 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.925577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.925577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.925711 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:56.925781 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:56.925885 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:19:56.925968 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:19:56.926476 30654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:56.926790 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.926710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:34960)
0925 15:19:56.926742 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.926770 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 15:19:56.927084 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.927031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:34976)
0925 15:19:56.927053 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.927072 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 15:19:56.927186 30654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:39269: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
W20260925 15:19:56.939559 30736 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:19:56.939752 30736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:56.939769 30736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:56.939816 30736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:56.940920 30736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:56.940968 30736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:56.940984 30736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:56.941008 30736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:56.941020 30736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:56.942479 30736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:39221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:40255,127.8.43.189:36193,127.8.43.188:39269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab
--principal=kudu/127.8.43.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.188:39269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.43.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:56.942658 30736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:56.942822 30736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:56.947527 30741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:56.947520 30742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:56.947520 30744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:56.947754 30736 server_base.cc:1034] running on GCE node
I20260925 15:19:56.948062 30736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:56.948367 30736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:56.949532 30736 hybrid_clock.cc:648] HybridClock initialized: now 1790349596949518 us; error 42 us; skew 500 ppm
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:56.952095 30736 init.cc:377] Logged in from keytab as kudu/127.8.43.188@KRBTEST.COM (short username kudu)
I20260925 15:19:56.952510 30736 webserver.cc:460] Webserver started at http://127.8.43.188:35839/ using document root <none> and password file <none>
I20260925 15:19:56.952728 30736 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:56.952793 30736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:56.952944 30736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:56.954015 30736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/data/instance:
uuid: "92eff0d143ab4460be01448996467d9a"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "c24ac3ae80bf9b264f4247282f4823e8"
server_key_iv: "a1e4f444c5e37ad95af4b1a6f7c779f0"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.954375 30736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/wal/instance:
uuid: "92eff0d143ab4460be01448996467d9a"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "c24ac3ae80bf9b264f4247282f4823e8"
server_key_iv: "a1e4f444c5e37ad95af4b1a6f7c779f0"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.955752 30736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:19:56.956475 30751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:56.956643 30736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:19:56.956739 30736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/wal
uuid: "92eff0d143ab4460be01448996467d9a"
format_stamp: "Formatted at 2026-09-25 15:19:56 on dist-test-slave-c5d6"
server_key: "c24ac3ae80bf9b264f4247282f4823e8"
server_key_iv: "a1e4f444c5e37ad95af4b1a6f7c779f0"
server_key_version: "encryptionkey@0"
I20260925 15:19:56.956821 30736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:56.958374 30654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } attempt: 1
I20260925 15:19:56.958797 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.958657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:34990)
0925 15:19:56.958721 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.958763 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:19:56.959169 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.959075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:35000)
0925 15:19:56.959123 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.959149 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:19:56.959295 30654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:39269: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
I20260925 15:19:56.969269 30729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } attempt: 2
I20260925 15:19:56.969659 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.969543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:35004)
0925 15:19:56.969600 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.969634 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:19:56.970042 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.969926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:35016)
0925 15:19:56.969988 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.970014 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:19:56.970158 30729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:39269: Network error: Client connection negotiation failed: client connection to 127.8.43.188:39269: connect: Connection refused (error 111)
I20260925 15:19:56.975374 30736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:56.977679 30736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:56.977824 30736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:56.981932 30736 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.188:39269
I20260925 15:19:56.981964 30803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.188:39269 every 8 connection(s)
I20260925 15:19:56.982323 30736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
I20260925 15:19:56.983376 30804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:56.984830 30804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:56.991896 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30736
I20260925 15:19:56.991992 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349306597827-8366-0/minicluster-data/master-2/wal/instance
I20260925 15:19:56.992175 8366 external_mini_cluster.cc:1428] Setting key e860e984aa95b10c65686d02056209c2
I20260925 15:19:56.993868 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.985448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.8.43.190:40255)
0925 15:19:56.985523 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.985526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.985760 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:19:56.986179 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.986180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.986189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.986240 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.986786 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.987314 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.988265 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.988449 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.988507 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.990595 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:56.990596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:56.990597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:56.990603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:56.992218 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.992803 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.992805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.992805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.992886 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:56.993186 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:56.993188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:56.993188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:56.993346 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:56.993431 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:56.993679 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 15:19:56.993762 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:19:56.994001 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.985437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:40255 (local address 127.0.0.1:48356)
0925 15:19:56.985644 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.985679 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.986100 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.986293 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.986295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.986523 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.986718 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.986721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.987391 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.987392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.988188 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.988190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.988531 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.988532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.988647 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.989113 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.989121 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.990488 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:56.992318 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.992324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.992342 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.992677 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.992992 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:56.992993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:56.992994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:56.993122 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:56.993417 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:56.993421 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:56.993530 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 15:19:56.993610 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:19:56.993720 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260925 15:19:56.994546 30804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
Sep 25 15:19:56 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:19:57.001371 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.994793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:36193 (local address 127.0.0.1:35604)
0925 15:19:56.994840 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:56.994850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:56.994887 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:56.995014 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:56.995015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:56.995145 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:56.995192 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.995193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.995853 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.995854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.996355 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:56.996356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.996752 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.996753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:56.996817 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.997196 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:56.997199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:56.998356 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:57.000229 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:57.000230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:57.000230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:57.000535 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:57.000771 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:57.000772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:57.000773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:57.000832 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:57.001056 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:57.001057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:57.001104 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:19:57.001188 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:19:57.001277 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:19:57.001439 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.994866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.8.43.189:36193)
0925 15:19:56.994918 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.994921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.994929 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:19:56.994953 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.994954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.994964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.995012 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.995307 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.995772 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.996447 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.996666 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.996752 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.998479 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:56.998480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:56.998481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:56.998488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.000113 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.000626 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.000627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.000628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.000690 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.000889 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.000889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.000889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.000992 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.001044 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.001205 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.001320 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:19:57.002022 30804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:57.002812 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:56.992577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.8.43.188:39269)
0925 15:19:56.992763 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:56.992768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:56.992836 (+ 68us) server_negotiation.cc:407] Connection header received
0925 15:19:56.992907 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:56.992909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:56.992921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:56.993015 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:56.993809 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.994972 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.995786 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:56.996094 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:56.996182 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:56.999066 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:56.999074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:56.999074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:56.999081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.001221 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.001746 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.001749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.001750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.001839 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.002164 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.002165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.002165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.002347 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.002504 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.002614 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.002699 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
Sep 25 15:19:57 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
Sep 25 15:19:57 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:57.009831 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.002274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:35022)
0925 15:19:57.002419 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:57.002429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:57.002461 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:57.002608 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:57.002608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:57.002771 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:57.002800 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.002802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.003616 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.003617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.004235 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.004237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.004699 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.004700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.004754 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.005092 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:57.005097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:57.006400 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:57.008828 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:57.008829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:57.008829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:57.008998 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:57.009213 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:57.009213 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:57.009214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:57.009267 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:57.009506 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:57.009507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:57.009543 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:19:57.009611 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:19:57.009705 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260925 15:19:57.009918 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.002303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.8.43.188:39269)
0925 15:19:57.002466 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.002468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.002479 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:57.002502 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.002503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.002516 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.002594 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.002870 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.003538 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.004315 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.004625 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.004679 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.006499 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:57.006500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:57.006500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:57.006505 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.008734 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.009078 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.009079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.009079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.009149 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.009337 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.009338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.009339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.009443 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.009513 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.009636 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.009754 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260925 15:19:57.011507 30804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a: Bootstrap starting.
I20260925 15:19:57.012107 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.003849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.8.43.190:40255)
0925 15:19:57.003891 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.003893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.003901 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:19:57.003923 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.003924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.003933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.003996 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.004511 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.005153 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.005823 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.006149 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.006211 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.008824 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:57.008826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:57.008826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:57.008833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.010914 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.011262 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.011263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.011263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.011340 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.011538 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.011539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.011539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.011670 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.011889 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.011927 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.012007 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:19:57.012152 30804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:57.012516 30804 log.cc:824] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:57.013397 30804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a: No bootstrap required, opened a new log
I20260925 15:19:57.015286 30804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } }
I20260925 15:19:57.015461 30804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92eff0d143ab4460be01448996467d9a, State: Initialized, Role: FOLLOWER
I20260925 15:19:57.015573 30804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } }
I20260925 15:19:57.015866 30811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.015959 30804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:57.015980 30811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 15:19:57 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
W20260925 15:19:57.017863 30822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:57.017962 30822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:57.018186 30736 master_runner.cc:186] Error getting master registration for 127.8.43.190:40255: OK, instance_id { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" instance_seqno: 1790349596689133 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:19:57.020586 30736 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:19:57.021068 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.012785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.8.43.189:36193)
0925 15:19:57.012838 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.012841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.012865 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:19:57.012921 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.012922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.012934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.013006 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.013596 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.014073 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.015003 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.015198 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.015345 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.018030 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:57.018032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:57.018032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:57.018039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.019815 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.020161 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.020162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.020163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.020220 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.020422 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.020423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.020423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.020537 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.020701 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.020893 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.020975 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:19:57.022526 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:57.023386 30628 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3cd067b8da6541eca56b9ea57bafd38d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48366
I20260925 15:19:57.034454 30654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } attempt: 2
Sep 25 15:19:57 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:19:57.041661 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.034761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:35028)
0925 15:19:57.034816 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:57.034828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:57.034902 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:57.035052 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:57.035053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:57.035212 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:57.035298 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.035300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.035980 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.035981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.036505 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.036506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.036934 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.036935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.036993 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.037470 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:57.037475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:57.038667 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:57.040538 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:57.040539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:57.040540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:57.040753 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:57.040960 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:57.040961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:57.040961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:57.041021 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:57.041343 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:57.041345 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:57.041389 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:19:57.041457 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:19:57.041554 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 15:19:57.041671 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.034839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.8.43.188:39269)
0925 15:19:57.034924 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.034926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.034955 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:19:57.034979 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.034979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.034989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.035039 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.035383 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.035886 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.036629 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.036858 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.036944 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.038751 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:57.038753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:57.038753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:57.038761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.040441 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.040823 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.040824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.040825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.040882 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.041084 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.041085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.041085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.041274 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.041345 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.041495 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.041577 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:19:57.043303 30654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d: Bootstrap starting.
I20260925 15:19:57.044030 30654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:57.044371 30654 log.cc:824] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:57.045241 30654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d: No bootstrap required, opened a new log
I20260925 15:19:57.046587 30654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } }
I20260925 15:19:57.046729 30654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd067b8da6541eca56b9ea57bafd38d, State: Initialized, Role: FOLLOWER
I20260925 15:19:57.046811 30654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } }
I20260925 15:19:57.047113 30654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:57.047089 30825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.047125 30628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:19:57.047185 30825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:57.047381 30628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:57.048144 30628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } }
I20260925 15:19:57.048452 30628 leader_election.cc:288] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [CANDIDATE]: Term 1 election: Requested vote from peers 6b465c19fed146de9457b7d67ab3f11a (127.8.43.189:36193), 92eff0d143ab4460be01448996467d9a (127.8.43.188:39269)
I20260925 15:19:57.048833 30778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cd067b8da6541eca56b9ea57bafd38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "92eff0d143ab4460be01448996467d9a"
I20260925 15:19:57.048956 30778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:57.049783 30778 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd067b8da6541eca56b9ea57bafd38d in term 1.
I20260925 15:19:57.050045 30603 leader_election.cc:302] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd067b8da6541eca56b9ea57bafd38d, 92eff0d143ab4460be01448996467d9a; no voters:
I20260925 15:19:57.050163 30825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:57.050307 30825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [term 1 LEADER]: Becoming Leader. State: Replica: 3cd067b8da6541eca56b9ea57bafd38d, State: Running, Role: LEADER
I20260925 15:19:57.050390 30825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } }
I20260925 15:19:57.050796 30841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd067b8da6541eca56b9ea57bafd38d. Latest consensus state: current_term: 1 leader_uuid: "3cd067b8da6541eca56b9ea57bafd38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.050887 30841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:57.049693 30586 master_runner.cc:186] Error getting master registration for 127.8.43.189:36193: OK, instance_id { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" instance_seqno: 1790349596826849 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:19:57.052263 30586 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:19:57.052542 30836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:57.052589 30836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:57.052663 30843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:57.053166 30843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:57.055610 30778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [term 1 FOLLOWER]: Refusing update from remote peer 3cd067b8da6541eca56b9ea57bafd38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:57.056021 30841 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:57.057331 30811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd067b8da6541eca56b9ea57bafd38d. Latest consensus state: current_term: 1 leader_uuid: "3cd067b8da6541eca56b9ea57bafd38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.057427 30811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:57.061285 30729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } attempt: 3
Sep 25 15:19:57 dist-test-slave-c5d6 krb5kdc[30573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:19:57.068539 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.053039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48376 (local address 127.8.43.190:40255)
0925 15:19:57.053073 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.053075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.053652 (+ 577us) server_negotiation.cc:407] Connection header received
0925 15:19:57.053750 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.053751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.053761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.053790 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.054571 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.055297 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.056482 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.056770 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.056811 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.058377 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:57.058379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:57.058379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:57.058386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.060164 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.060724 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.060725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.060725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.060783 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.060987 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.060988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.060988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.061124 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.061220 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.068340 (+ 7120us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.068419 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:19:57.071296 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.061523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:39269 (local address 127.0.0.1:35040)
0925 15:19:57.061615 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:57.061626 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:57.061663 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:57.061912 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:57.061913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:57.062068 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:57.062107 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.062108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.062928 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.062929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.063711 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:57.063714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.064687 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.064688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:57.064749 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.065190 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:57.065194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:57.067207 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:57.070047 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:57.070048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:57.070048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:57.070284 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:57.070580 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:57.070581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:57.070581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:57.070650 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:57.070955 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:57.070956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:57.071009 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:19:57.071071 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:19:57.071161 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 15:19:57.071703 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.061765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.8.43.188:39269)
0925 15:19:57.061801 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.061802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.061809 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:19:57.061824 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.061825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.061833 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.061877 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.062184 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.062847 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.063802 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.064387 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.064615 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.067904 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:57.067905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:57.067905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:57.067912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.069960 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.070400 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.070401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.070401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.070490 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.070725 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.070726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.070726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.070858 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.070915 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.071534 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.071615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:19:57.072286 30825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cd067b8da6541eca56b9ea57bafd38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.072399 30825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:57.072639 30848 mvcc.cc:205] Tried to move back new op lower bound from 7333271949536092160 to 7333271949519040512. Current Snapshot: MvccSnapshot[applied={T|T < 7333271949536092160}]
I20260925 15:19:57.073050 30843 catalog_manager.cc:1347] Generated new cluster ID: a71bdcbdb49b4cec9b1e112f1702f517
I20260925 15:19:57.073102 30843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:19:57.073360 30729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a: Bootstrap starting.
I20260925 15:19:57.074000 30729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:57.074255 30729 log.cc:824] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:57.074780 30811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cd067b8da6541eca56b9ea57bafd38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.074862 30811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:57.074927 30849 mvcc.cc:205] Tried to move back new op lower bound from 7333271949536092160 to 7333271949519040512. Current Snapshot: MvccSnapshot[applied={T|T < 7333271949536092160}]
I20260925 15:19:57.075541 30729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a: No bootstrap required, opened a new log
I20260925 15:19:57.077310 30729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } }
I20260925 15:19:57.077431 30729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b465c19fed146de9457b7d67ab3f11a, State: Initialized, Role: FOLLOWER
I20260925 15:19:57.077500 30729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } }
I20260925 15:19:57.077769 30729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:57.077827 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cd067b8da6541eca56b9ea57bafd38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6b465c19fed146de9457b7d67ab3f11a"
I20260925 15:19:57.077944 30703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:57.078768 30703 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cd067b8da6541eca56b9ea57bafd38d in term 1.
I20260925 15:19:57.077773 30854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.079326 30854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:57.079404 30702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [term 1 FOLLOWER]: Refusing update from remote peer 3cd067b8da6541eca56b9ea57bafd38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:57.079670 30841 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:57.081640 30866 mvcc.cc:205] Tried to move back new op lower bound from 7333271949536092160 to 7333271949519040512. Current Snapshot: MvccSnapshot[applied={T|T < 7333271949536092160}]
I20260925 15:19:57.083786 30868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cd067b8da6541eca56b9ea57bafd38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.083916 30868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:57.086369 30841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cd067b8da6541eca56b9ea57bafd38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.086462 30841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:57.086719 30854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd067b8da6541eca56b9ea57bafd38d. Latest consensus state: current_term: 1 leader_uuid: "3cd067b8da6541eca56b9ea57bafd38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd067b8da6541eca56b9ea57bafd38d" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 40255 } } peers { permanent_uuid: "6b465c19fed146de9457b7d67ab3f11a" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 36193 } } peers { permanent_uuid: "92eff0d143ab4460be01448996467d9a" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 39269 } } }
I20260925 15:19:57.086788 30854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:57.089375 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.053304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.8.43.189:36193)
0925 15:19:57.053381 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.053383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.059350 (+ 5967us) server_negotiation.cc:407] Connection header received
0925 15:19:57.059468 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.059469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.059481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.059538 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.060674 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.061393 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.063205 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.063543 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.063586 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.065861 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:57.065862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:57.065863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:57.065882 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.067807 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.088506 (+ 20699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.088507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.088508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.088573 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.088787 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.088788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.088788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.088913 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.088987 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.089202 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.089286 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:19:57.090018 30843 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:57.090728 30843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:57.090813 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:57.052859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.8.43.188:39269)
0925 15:19:57.052900 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.052914 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.060063 (+ 7149us) server_negotiation.cc:407] Connection header received
0925 15:19:57.061070 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.061071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.061081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.061109 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.061739 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.062449 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.067327 (+ 4878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:57.067643 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:57.067710 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:57.087542 (+ 19832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:57.087544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:57.087544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:57.087551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:57.089316 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.089933 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.089934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.089934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.089990 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:57.090229 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:57.090230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:57.090230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:57.090353 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:57.090455 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:57.090646 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 15:19:57.090725 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:19:57.095909 30865 catalog_manager.cc:1259] Loaded cluster ID: a71bdcbdb49b4cec9b1e112f1702f517
I20260925 15:19:57.095999 30865 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a: loading cluster ID for follower catalog manager: success
I20260925 15:19:57.097846 30865 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a: acquiring CA information for follower catalog manager: success
W20260925 15:19:57.098031 30865 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260925 15:19:57.106760 30843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cd067b8da6541eca56b9ea57bafd38d: Generated new TSK 0
I20260925 15:19:57.106930 30843 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:19:57.131057 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.129454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.8.43.189:36193)
0925 15:19:57.129496 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.129499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.130399 (+ 900us) server_negotiation.cc:407] Connection header received
0925 15:19:57.130524 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.130526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.130537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.130568 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.130986 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35622: BlockingRecv error: recv got EOF from 127.0.0.1:35622 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:19:57.131846 30658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.129826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.8.43.190:40255)
0925 15:19:57.129867 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.129869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.131333 (+ 1464us) server_negotiation.cc:407] Connection header received
0925 15:19:57.131442 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.131444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.131455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.131485 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.131785 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48382: BlockingRecv error: recv got EOF from 127.0.0.1:48382 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:19:57.132607 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.130145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.8.43.188:39269)
0925 15:19:57.130185 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.130187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.132070 (+ 1883us) server_negotiation.cc:407] Connection header received
0925 15:19:57.132198 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.132199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.132209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.132239 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.132548 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35048: BlockingRecv error: recv got EOF from 127.0.0.1:35048 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:19:57.134974 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.134169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.8.43.189:36193)
0925 15:19:57.134213 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.134215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.134466 (+ 251us) server_negotiation.cc:407] Connection header received
0925 15:19:57.134574 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.134575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.134593 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.134622 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.134928 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35636: BlockingRecv error: recv got EOF from 127.0.0.1:35636 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 15:19:57.137962 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.137184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.8.43.189:36193)
0925 15:19:57.137217 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.137218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.137461 (+ 243us) server_negotiation.cc:407] Connection header received
0925 15:19:57.137555 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.137556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.137565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.137594 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.137919 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35646: BlockingRecv error: recv got EOF from 127.0.0.1:35646 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 15:19:57.141680 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.140945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.8.43.189:36193)
0925 15:19:57.140977 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.140980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.141207 (+ 227us) server_negotiation.cc:407] Connection header received
0925 15:19:57.141297 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.141298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.141307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.141336 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.141639 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35662: BlockingRecv error: recv got EOF from 127.0.0.1:35662 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 15:19:57.147521 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.146717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.8.43.189:36193)
0925 15:19:57.146749 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.146751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.146985 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:19:57.147081 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.147081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.147091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.147119 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.147476 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35668: BlockingRecv error: recv got EOF from 127.0.0.1:35668 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 15:19:57.157275 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.156523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.8.43.189:36193)
0925 15:19:57.156567 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.156569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.156801 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:19:57.156892 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.156893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.156903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.156932 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.157233 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35674: BlockingRecv error: recv got EOF from 127.0.0.1:35674 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 15:19:57.175319 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.174393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.8.43.189:36193)
0925 15:19:57.174444 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.174446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.174725 (+ 279us) server_negotiation.cc:407] Connection header received
0925 15:19:57.174827 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.174828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.174840 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.174870 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.175264 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35690: BlockingRecv error: recv got EOF from 127.0.0.1:35690 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 15:19:57.209383 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.208606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.8.43.189:36193)
0925 15:19:57.208655 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.208657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.208794 (+ 137us) server_negotiation.cc:407] Connection header received
0925 15:19:57.208879 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.208880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.208894 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.208989 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.209327 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35704: BlockingRecv error: recv got EOF from 127.0.0.1:35704 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 15:19:57.275313 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.274501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.8.43.189:36193)
0925 15:19:57.274575 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.274578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.274705 (+ 127us) server_negotiation.cc:407] Connection header received
0925 15:19:57.274791 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.274792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.274804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.274869 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.275252 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35710: BlockingRecv error: recv got EOF from 127.0.0.1:35710 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 15:19:57.405220 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.404532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.8.43.189:36193)
0925 15:19:57.404599 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.404601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.404612 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:57.404697 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.404698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.404715 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.404782 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.405152 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35724: BlockingRecv error: recv got EOF from 127.0.0.1:35724 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 15:19:57.663292 30733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:57.662536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.8.43.189:36193)
0925 15:19:57.662606 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:57.662608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:57.662638 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:19:57.662718 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:57.662719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:57.662729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:57.662792 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:57.663164 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35730: BlockingRecv error: recv got EOF from 127.0.0.1:35730 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:19:58.018457 30822 catalog_manager.cc:1259] Loaded cluster ID: a71bdcbdb49b4cec9b1e112f1702f517
I20260925 15:19:58.018534 30822 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a: loading cluster ID for follower catalog manager: success
I20260925 15:19:58.019397 30822 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a: acquiring CA information for follower catalog manager: success
I20260925 15:19:58.019672 30822 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92eff0d143ab4460be01448996467d9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:19:58.098793 30865 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b465c19fed146de9457b7d67ab3f11a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:19:58.179508 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:58.176445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.8.43.189:36193)
0925 15:19:58.176605 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:58.176608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:58.176618 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:19:58.176657 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:58.176657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:58.176709 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:58.176791 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:58.176991 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.177671 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.178593 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.178802 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.178908 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.179107 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:58.179233 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:58.179296 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:58.179338 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 15:19:58.179356 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260925 15:19:58.184237 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:58.181553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.8.43.188:39269)
0925 15:19:58.181654 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:58.181657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:58.181682 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:19:58.181770 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:58.181771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:58.181817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:58.181929 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:58.182073 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.182564 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.183381 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.183604 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.183688 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.183865 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:58.183986 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:58.184077 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:58.184120 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:19:58.184144 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
I20260925 15:19:58.188719 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:58.185926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.8.43.188:39269)
0925 15:19:58.185976 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:58.185977 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:58.186129 (+ 152us) server_negotiation.cc:407] Connection header received
0925 15:19:58.186234 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:58.186235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:58.186247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:58.186304 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:58.186495 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.187179 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.187957 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.188232 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.188287 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.188456 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:58.188486 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:58.188546 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:58.188618 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:19:58.188641 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:19:58.189160 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:58.185882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.8.43.189:36193)
0925 15:19:58.185991 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:58.185993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:58.186112 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:19:58.186186 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:58.186187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:58.186199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:58.186252 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:58.186386 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.187147 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.188311 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.188606 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.188670 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.188868 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:58.188902 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:58.188936 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:58.189059 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 15:19:58.189083 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:19:58.189184 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:58.185892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.8.43.190:40255)
0925 15:19:58.186048 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:58.186050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:58.186087 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:19:58.186191 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:58.186193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:58.186243 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:58.186321 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:19:58.186479 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.186985 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.188104 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:58.188408 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:58.188476 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:58.188733 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:19:58.188844 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:19:58.188897 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:58.189067 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:19:58.189096 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":59,"threads_started":1}
I20260925 15:19:58.190778 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30586
I20260925 15:19:58.196089 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30661
I20260925 15:19:58.200891 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30736
2026-09-25T15:19:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2796 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:19:58 dist-test-slave-c5d6 krb5kdc[30953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:19:58 dist-test-slave-c5d6 krb5kdc[30953](info): set up 2 sockets
Sep 25 15:19:58 dist-test-slave-c5d6 krb5kdc[30953](info): commencing operation
krb5kdc: starting...
W20260925 15:19:59.231392 8366 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/test-user.keytab.
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.43.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
Entry for principal kudu/127.8.43.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab.
I20260925 15:19:59.352017 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38927
--webserver_interface=127.8.43.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:38927,127.8.43.189:33525,127.8.43.188:36311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:19:59.431126 30966 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:19:59.431345 30966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:59.431362 30966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:59.431382 30966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:59.432565 30966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:59.432603 30966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:59.432617 30966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:59.432629 30966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:59.432643 30966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:59.434166 30966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:38927,127.8.43.189:33525,127.8.43.188:36311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.190.keytab
--principal=kudu/127.8.43.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.190:38927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.43.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:59.434412 30966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:59.434649 30966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:59.439358 30971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:59.439352 30974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:59.439352 30972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:59.439402 30966 server_base.cc:1034] running on GCE node
I20260925 15:19:59.439764 30966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:59.440060 30966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:59.441200 30966 hybrid_clock.cc:648] HybridClock initialized: now 1790349599441179 us; error 37 us; skew 500 ppm
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:59.443614 30966 init.cc:377] Logged in from keytab as kudu/127.8.43.190@KRBTEST.COM (short username kudu)
I20260925 15:19:59.444083 30966 webserver.cc:460] Webserver started at http://127.8.43.190:36583/ using document root <none> and password file <none>
I20260925 15:19:59.444305 30966 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:59.444368 30966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:59.444514 30966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:59.445550 30966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/data/instance:
uuid: "7fc4c6b562f34fc3a268573b891cf5bc"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "c407324ba4fb5744a39bc476f544e42a"
server_key_iv: "afa548135002a9d6af1e7b038c13f6e3"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.445900 30966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/wal/instance:
uuid: "7fc4c6b562f34fc3a268573b891cf5bc"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "c407324ba4fb5744a39bc476f544e42a"
server_key_iv: "afa548135002a9d6af1e7b038c13f6e3"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.447165 30966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:19:59.447944 30981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:59.448135 30966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:59.448231 30966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/wal
uuid: "7fc4c6b562f34fc3a268573b891cf5bc"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "c407324ba4fb5744a39bc476f544e42a"
server_key_iv: "afa548135002a9d6af1e7b038c13f6e3"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.448310 30966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:59.472339 30966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:59.474673 30966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:59.474834 30966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:59.478852 30966 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.190:38927
I20260925 15:19:59.478888 31033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.190:38927 every 8 connection(s)
I20260925 15:19:59.479252 30966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/data/info.pb
I20260925 15:19:59.480238 31034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:59.481721 31034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:59.486552 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 30966
I20260925 15:19:59.486646 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-0/wal/instance
I20260925 15:19:59.486838 8366 external_mini_cluster.cc:1428] Setting key ee2d18618ed17d6e89b1ee5cdf6ece00
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:59.490998 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.482343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38927 (local address 127.0.0.1:41356)
0925 15:19:59.482496 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.482529 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.482934 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.483206 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.483208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.483493 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.483642 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.483646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.484634 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.484635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.485294 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.485296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.485767 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.485768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.485820 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.486245 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.486253 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.487831 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.489557 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.489563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.489565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.489755 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.489995 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.489996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.489997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.490085 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.490426 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.490431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.490489 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.490598 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.490826 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":95,"thread_start_us":71,"threads_started":1}
I20260925 15:19:59.491001 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.482492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.8.43.190:38927)
0925 15:19:59.482620 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.482624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.482662 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:19:59.483088 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.483090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.483106 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.483180 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.483734 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.484530 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.485369 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.485677 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.485752 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.487921 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.487928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.487929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.487935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.489455 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.489844 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.489847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.489847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.489925 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.490152 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.490153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.490154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.490354 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.490454 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.490630 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.490762 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260925 15:19:59.491794 31034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:59.492144 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.492023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41050)
0925 15:19:59.492071 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.492115 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:19:59.492271 30982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
I20260925 15:19:59.492703 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.492613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41066)
0925 15:19:59.492660 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.492683 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:19:59.492815 31034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:33525: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.43.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab.
Entry for principal kudu/127.8.43.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab.
I20260925 15:19:59.510851 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.189:33525
--webserver_interface=127.8.43.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab
--principal=kudu/127.8.43.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:38927,127.8.43.189:33525,127.8.43.188:36311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:59.542038 31034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } attempt: 1
I20260925 15:19:59.542672 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.542535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41076)
0925 15:19:59.542604 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.542645 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:19:59.543072 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.542960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41090)
0925 15:19:59.543022 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.543051 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:19:59.543255 31034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:33525: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
W20260925 15:19:59.590927 31041 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:19:59.591127 31041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:59.591143 31041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:59.591162 31041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:59.592355 31041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:59.592396 31041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:59.592410 31041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:59.592422 31041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:59.592476 31041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:59.593923 31041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:38927,127.8.43.189:33525,127.8.43.188:36311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.189.keytab
--principal=kudu/127.8.43.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.189:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.43.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:59.594172 31041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:59.594390 31041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:59.599040 31047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:59.599131 31041 server_base.cc:1034] running on GCE node
W20260925 15:19:59.599020 31049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:59.599018 31046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:59.599522 31041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:59.599805 31041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:59.600970 31041 hybrid_clock.cc:648] HybridClock initialized: now 1790349599600960 us; error 29 us; skew 500 ppm
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:59.603477 31041 init.cc:377] Logged in from keytab as kudu/127.8.43.189@KRBTEST.COM (short username kudu)
I20260925 15:19:59.603854 31041 webserver.cc:460] Webserver started at http://127.8.43.189:33941/ using document root <none> and password file <none>
I20260925 15:19:59.604033 31041 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:59.604072 31041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:59.604158 31041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:59.605165 31041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/data/instance:
uuid: "a9f29ad8dfa5449995eec57374e962bb"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "03cb6d08e8f6eb778eb37f6d4e9d274e"
server_key_iv: "b4871c073fbc109667cc7c014faba2ab"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.605506 31041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/wal/instance:
uuid: "a9f29ad8dfa5449995eec57374e962bb"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "03cb6d08e8f6eb778eb37f6d4e9d274e"
server_key_iv: "b4871c073fbc109667cc7c014faba2ab"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.606746 31041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:59.607458 31056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:59.607638 31041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:59.607743 31041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/wal
uuid: "a9f29ad8dfa5449995eec57374e962bb"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "03cb6d08e8f6eb778eb37f6d4e9d274e"
server_key_iv: "b4871c073fbc109667cc7c014faba2ab"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.607817 31041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:59.611433 31034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } attempt: 2
I20260925 15:19:59.611822 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.611697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41096)
0925 15:19:59.611751 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.611798 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:19:59.612178 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.612093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41112)
0925 15:19:59.612136 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.612160 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:19:59.612303 31034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.189:33525: Network error: Client connection negotiation failed: client connection to 127.8.43.189:33525: connect: Connection refused (error 111)
I20260925 15:19:59.616313 31041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:59.618500 31041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:59.618667 31041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:59.622620 31041 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.189:33525
I20260925 15:19:59.622690 31108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.189:33525 every 8 connection(s)
I20260925 15:19:59.622995 31041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/data/info.pb
I20260925 15:19:59.623961 31109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:59.625425 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 31041
I20260925 15:19:59.625506 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-1/wal/instance
I20260925 15:19:59.625444 31109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:59.625715 8366 external_mini_cluster.cc:1428] Setting key 29e14722c2dcc15da499554764b70d64
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.43.188@KRBTEST.COM; defaulting to no policy
I20260925 15:19:59.634891 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.626181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.8.43.190:38927)
0925 15:19:59.626221 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.626223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.626404 (+ 181us) server_negotiation.cc:407] Connection header received
0925 15:19:59.626888 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.626889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.626904 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.626977 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.627612 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.628345 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.629335 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.629622 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.629711 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.631837 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.631839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.631839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.631847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.633412 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.633843 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.633844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.633844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.633901 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.634162 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.634163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.634163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.634369 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.634435 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.634708 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.634802 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:19:59.635076 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.626161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38927 (local address 127.0.0.1:41368)
0925 15:19:59.626301 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.626333 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.626789 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.627024 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.627027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.627321 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.627515 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.627519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.628444 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.628445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.629222 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.629224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.629737 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.629739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.629857 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.630359 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.630367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.631728 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.633520 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.633527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.633528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.633750 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.633978 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.633979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.633979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.634085 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.634429 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.634435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.634577 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.634688 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.634849 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260925 15:19:59.635713 31109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.43.188@KRBTEST.COM" created.
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:19:59.643291 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.636107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41114)
0925 15:19:59.636160 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.636171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.636216 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.636492 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.636493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.636624 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.636666 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.636667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.637501 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.637502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.638008 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.638009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.638506 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.638507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.638573 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.638952 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.638957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.640354 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.642210 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.642211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.642211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.642375 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.642612 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.642613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.642614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.642665 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.642966 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.642967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.643003 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.643065 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.643161 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:19:59.643347 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.636202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.8.43.189:33525)
0925 15:19:59.636333 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.636337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.636353 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:19:59.636372 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.636373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.636392 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.636478 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.636738 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.637413 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.638100 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.638414 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.638507 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.640459 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.640467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.640468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.640474 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.642106 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.642474 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.642477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.642477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.642534 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.642743 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.642744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.642744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.642879 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.642985 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.643092 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.643211 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260925 15:19:59.644205 31109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:59.644543 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.644420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45446)
0925 15:19:59.644473 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.644518 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:19:59.644693 31058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
I20260925 15:19:59.645292 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.645053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45462)
0925 15:19:59.645121 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.645148 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:19:59.645547 31109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:36311: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.43.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab.
Entry for principal kudu/127.8.43.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab.
I20260925 15:19:59.652004 8366 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcJTRKe/build/release/bin/kudu
/tmp/dist-test-taskcJTRKe/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.188:36311
--webserver_interface=127.8.43.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab
--principal=kudu/127.8.43.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.43.148:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.43.190:38927,127.8.43.189:33525,127.8.43.188:36311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:19:59.694793 31109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } attempt: 1
I20260925 15:19:59.695322 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.695171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45478)
0925 15:19:59.695253 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.695293 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:19:59.695751 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.695617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45482)
0925 15:19:59.695681 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.695722 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:19:59.695864 31109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:36311: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
I20260925 15:19:59.703464 31034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } attempt: 3
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:19:59.710649 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.703826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.8.43.189:33525)
0925 15:19:59.703883 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.703886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.703897 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:19:59.703991 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.703992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.704005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.704075 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.704401 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.704902 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.705749 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.705970 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.706075 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.707964 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.707966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.707966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.707974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.709534 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.709853 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.709854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.709854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.709933 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.710135 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.710136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.710136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.710297 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.710355 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.710530 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.710592 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:19:59.710695 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.703762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41128)
0925 15:19:59.703819 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.703837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.703906 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.704080 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.704081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.704245 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.704309 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.704310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.705044 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.705045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.705653 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.705656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.706074 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.706075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.706159 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.706609 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.706622 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.707883 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.709614 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.709614 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.709615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.709793 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.710007 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.710008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.710009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.710072 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.710365 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.710365 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.710416 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.710503 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.710600 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:19:59.711053 31034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:59.711380 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.711294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45494)
0925 15:19:59.711325 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.711356 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 15:19:59.711742 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.711649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45506)
0925 15:19:59.711698 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.711723 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:19:59.711854 31034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:36311: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
W20260925 15:19:59.732517 31116 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:19:59.732697 31116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:19:59.732714 31116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:19:59.732759 31116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:19:59.733882 31116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:19:59.733935 31116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:19:59.733956 31116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:19:59.734005 31116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:19:59.734026 31116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:19:59.735530 31116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.43.148:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.43.190:38927,127.8.43.189:33525,127.8.43.188:36311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/krb5kdc/kudu_127.8.43.188.keytab
--principal=kudu/127.8.43.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.43.188:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.43.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:19:59.735764 31116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:19:59.735980 31116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:19:59.740542 31124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:19:59.740592 31122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:59.740624 31116 server_base.cc:1034] running on GCE node
W20260925 15:19:59.740556 31121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:19:59.740973 31116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:19:59.741201 31116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:19:59.742375 31116 hybrid_clock.cc:648] HybridClock initialized: now 1790349599742361 us; error 24 us; skew 500 ppm
I20260925 15:19:59.743013 31034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } attempt: 1
I20260925 15:19:59.743388 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.743290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45510)
0925 15:19:59.743331 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.743363 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 15:19:59.743737 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.743659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45514)
0925 15:19:59.743701 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.743723 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:19:59.743850 31034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:36311: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:19:59.745076 31116 init.cc:377] Logged in from keytab as kudu/127.8.43.188@KRBTEST.COM (short username kudu)
I20260925 15:19:59.745455 31116 webserver.cc:460] Webserver started at http://127.8.43.188:45515/ using document root <none> and password file <none>
I20260925 15:19:59.745637 31116 fs_manager.cc:359] Metadata directory not provided
I20260925 15:19:59.745676 31116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:19:59.745761 31116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:19:59.746790 31116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/data/instance:
uuid: "2c43db0f41fe423592cd6b2006e99d82"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "21337aa4a1c718dc05c91ff38399ab86"
server_key_iv: "0a059b8bf5a7e555a1314c33712b5af9"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.747073 31116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/wal/instance:
uuid: "2c43db0f41fe423592cd6b2006e99d82"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "21337aa4a1c718dc05c91ff38399ab86"
server_key_iv: "0a059b8bf5a7e555a1314c33712b5af9"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.748351 31116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:19:59.749097 31131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:19:59.749408 31116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:19:59.749521 31116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/data,/tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/wal
uuid: "2c43db0f41fe423592cd6b2006e99d82"
format_stamp: "Formatted at 2026-09-25 15:19:59 on dist-test-slave-c5d6"
server_key: "21337aa4a1c718dc05c91ff38399ab86"
server_key_iv: "0a059b8bf5a7e555a1314c33712b5af9"
server_key_version: "encryptionkey@0"
I20260925 15:19:59.749594 31116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:19:59.764011 31109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } attempt: 2
I20260925 15:19:59.764442 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.764320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45524)
0925 15:19:59.764376 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.764415 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:19:59.764801 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.764707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45536)
0925 15:19:59.764751 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.764775 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:19:59.764899 31109 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.43.188:36311: Network error: Client connection negotiation failed: client connection to 127.8.43.188:36311: connect: Connection refused (error 111)
I20260925 15:19:59.765952 31116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:19:59.768313 31116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:19:59.768479 31116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:19:59.772413 31116 rpc_server.cc:307] RPC server started. Bound to: 127.8.43.188:36311
I20260925 15:19:59.772449 31183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.43.188:36311 every 8 connection(s)
I20260925 15:19:59.772796 31116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/data/info.pb
I20260925 15:19:59.773789 31184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:19:59.775430 31184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:59.776494 8366 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcJTRKe/build/release/bin/kudu as pid 31116
I20260925 15:19:59.776607 8366 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcJTRKe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349306597827-8366-0/minicluster-data/master-2/wal/instance
I20260925 15:19:59.776790 8366 external_mini_cluster.cc:1428] Setting key 0b19508e8bed32f62fe335d9a9b381ac
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:19:59.784788 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.776144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.8.43.190:38927)
0925 15:19:59.776233 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.776236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.776441 (+ 205us) server_negotiation.cc:407] Connection header received
0925 15:19:59.777038 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.777038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.777052 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.777108 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.777840 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.778438 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.779407 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.779594 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.779647 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.781925 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.781926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.781927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.781934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.783413 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.783836 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.783837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.783838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.783899 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.784145 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.784146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.784147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.784267 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.784340 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.784606 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.784687 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:19:59.785005 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.776128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.190:38927 (local address 127.0.0.1:41382)
0925 15:19:59.776297 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.776340 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.776916 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.777181 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.777185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.777469 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.777747 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.777758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.778521 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.778522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.779330 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.779333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.779690 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.779691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.779827 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.780386 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.780394 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.781826 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.783514 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.783521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.783522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.783753 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.783964 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.783965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.783965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.784079 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.784333 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.784339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.784476 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.784584 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.784760 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260925 15:19:59.785568 31184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:59.786381 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.777324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.8.43.188:36311)
0925 15:19:59.777501 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.777506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.777525 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:19:59.777545 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.777547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.777565 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.777659 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.778231 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.779134 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.780166 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.780539 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.780598 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.783195 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.783202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.783202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.783209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.785027 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.785524 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.785527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.785528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.785598 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.785803 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.785804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.785804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.785981 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.786098 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.786201 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.786290 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.190@KRBTEST.COM
I20260925 15:19:59.792672 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.785820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.189:33525 (local address 127.0.0.1:41144)
0925 15:19:59.785853 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.785866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.785900 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.786135 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.786136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.786316 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.786351 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.786353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.787160 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.787161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.787801 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.787803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.788185 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.788186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.788243 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.788601 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.788606 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.789928 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.791544 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.791545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.791546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.791755 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.792006 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.792007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.792007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.792063 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.792332 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.792333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.792372 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.792497 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.792583 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 15:19:59.792672 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.785933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.8.43.189:33525)
0925 15:19:59.785999 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.786001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.786007 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:19:59.786031 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.786032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.786041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.786089 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.786414 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.787045 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.787887 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.788105 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.788175 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.790036 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.790037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.790038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.790045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.791439 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.791845 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.791847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.791847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.791902 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.792134 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.792134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.792135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.792247 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.792328 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.792524 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.792591 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:19:59.793085 31184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:19:59.794772 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.787321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.8.43.190:38927)
0925 15:19:59.787358 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.787360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.787376 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:19:59.787423 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.787423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.787432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.787493 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.787912 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.788384 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.789297 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.789621 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.789695 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.792042 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.792043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.792043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.792050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.793760 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.794106 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.794107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.794107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.794152 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.794331 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.794332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.794332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.794413 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.794526 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.794618 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.794686 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.188@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.43.189@KRBTEST.COM
I20260925 15:19:59.800216 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.793359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45554)
0925 15:19:59.793400 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.793409 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.793442 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.793553 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.793553 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.793692 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.793726 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.793727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.794608 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.794609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.795510 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.795513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.795931 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.795932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.795991 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.796353 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.796357 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.797387 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.799110 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.799110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.799111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.799327 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.799520 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.799520 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.799521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.799586 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.799851 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.799852 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.799883 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.799958 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.800053 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 15:19:59.800262 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.793399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.8.43.188:36311)
0925 15:19:59.793431 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.793433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.793440 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:19:59.793492 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.793492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.793501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.793547 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.793804 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.794530 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.795587 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.795858 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.795912 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.797467 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.797468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.797468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.797475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.799030 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.799391 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.799392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.799393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.799446 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.799650 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.799651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.799652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.799783 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.799856 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.799997 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.800093 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":15}
I20260925 15:19:59.801807 31184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82: Bootstrap starting.
I20260925 15:19:59.802721 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.795410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.8.43.189:33525)
0925 15:19:59.795441 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.795451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.795459 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:19:59.795501 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.795501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.795510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.795566 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.796072 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.796633 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.797305 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.797515 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.797591 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.799963 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.799964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.799964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.799970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.801626 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.802011 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.802012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.802012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.802076 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.802312 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.802313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.802313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.802406 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.802509 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.802613 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.802667 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:19:59.802842 31184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:59.803284 31184 log.cc:824] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:59.804144 31184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82: No bootstrap required, opened a new log
I20260925 15:19:59.804244 8366 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:19:59.805861 31184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } }
I20260925 15:19:59.805982 31184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c43db0f41fe423592cd6b2006e99d82, State: Initialized, Role: FOLLOWER
I20260925 15:19:59.805295 31008 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7fc4c6b562f34fc3a268573b891cf5bc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41384
I20260925 15:19:59.806087 31184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } }
I20260925 15:19:59.806393 31190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.806433 31184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:59.806502 31190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:19:59.808346 31201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:59.808399 31201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:59.808558 31116 master_runner.cc:186] Error getting master registration for 127.8.43.190:38927: OK, instance_id { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" instance_seqno: 1790349599477741 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:19:59.810771 31116 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:19:59.818945 31034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } attempt: 2
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.190@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:19:59.826066 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.819244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45558)
0925 15:19:59.819294 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.819304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.819345 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.819462 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.819463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.819620 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.819660 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.819661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.820339 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.820341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.820854 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.820856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.821263 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.821265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.821326 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.821786 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.821791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.823327 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.824984 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.824985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.824986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.825202 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.825397 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.825397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.825398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.825463 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.825748 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.825749 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.825796 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.825866 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.825955 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 15:19:59.826077 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.819303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.8.43.188:36311)
0925 15:19:59.819354 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.819356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.819368 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:19:59.819392 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.819393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.819410 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.819461 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.819746 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.820265 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.820963 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.821198 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.821255 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.823424 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.823425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.823426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.823432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.824903 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.825275 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.825276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.825276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.825331 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.825533 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.825533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.825534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.825680 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.825742 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.825898 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.825967 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:19:59.827556 31034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc: Bootstrap starting.
I20260925 15:19:59.828236 31034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:59.828513 31034 log.cc:824] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:59.829211 31034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc: No bootstrap required, opened a new log
I20260925 15:19:59.830180 31034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } }
I20260925 15:19:59.830317 31034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc4c6b562f34fc3a268573b891cf5bc, State: Initialized, Role: FOLLOWER
I20260925 15:19:59.830410 31034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } }
I20260925 15:19:59.830648 31204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.830703 31034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:59.830768 31204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:59.830775 31008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:19:59.830962 31008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:59.831777 31008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } }
I20260925 15:19:59.832484 31158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2c43db0f41fe423592cd6b2006e99d82"
I20260925 15:19:59.832626 31158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:59.833443 31158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fc4c6b562f34fc3a268573b891cf5bc in term 1.
I20260925 15:19:59.833622 31008 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [CANDIDATE]: Term 1 election: Requested vote from peers a9f29ad8dfa5449995eec57374e962bb (127.8.43.189:33525), 2c43db0f41fe423592cd6b2006e99d82 (127.8.43.188:36311)
I20260925 15:19:59.833657 30983 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c43db0f41fe423592cd6b2006e99d82, 7fc4c6b562f34fc3a268573b891cf5bc; no voters:
I20260925 15:19:59.833776 31204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:19:59.833894 31204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [term 1 LEADER]: Becoming Leader. State: Replica: 7fc4c6b562f34fc3a268573b891cf5bc, State: Running, Role: LEADER
I20260925 15:19:59.834025 31204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } }
I20260925 15:19:59.834450 31204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc4c6b562f34fc3a268573b891cf5bc. Latest consensus state: current_term: 1 leader_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.834511 31204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:59.834868 31222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:19:59.835714 31222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:19:59.833495 30966 master_runner.cc:186] Error getting master registration for 127.8.43.189:33525: OK, instance_id { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" instance_seqno: 1790349599621554 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:19:59.836295 30966 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:19:59.837980 31158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [term 1 FOLLOWER]: Refusing update from remote peer 7fc4c6b562f34fc3a268573b891cf5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:59.838322 31216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:19:59.839601 31190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc4c6b562f34fc3a268573b891cf5bc. Latest consensus state: current_term: 1 leader_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.839695 31190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:59.844062 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.834698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.8.43.188:36311)
0925 15:19:59.834734 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.834736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.835478 (+ 742us) server_negotiation.cc:407] Connection header received
0925 15:19:59.835639 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.835640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.835649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.835676 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.836399 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.837104 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.838162 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.838462 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.838534 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.840065 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.840066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.840066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.840073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.841619 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.842483 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.842484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.842484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.842542 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.843101 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.843102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.843103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.843284 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.843377 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.843779 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.843916 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:19:59.848345 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.836413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.8.43.189:33525)
0925 15:19:59.836456 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.836458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.839356 (+ 2898us) server_negotiation.cc:407] Connection header received
0925 15:19:59.839463 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.839464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.839476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.839508 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.840131 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.841301 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.842526 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.842833 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.842872 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.844483 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.844484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.844484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.844503 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.846679 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.847341 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.847343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.847343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.847425 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.847680 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.847681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.847681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.847807 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.847884 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.848154 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.848254 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:19:59.851512 31216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.851629 31216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: This master's current role is: LEADER
I20260925 15:19:59.851637 31190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.851722 31190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:59.851944 31228 mvcc.cc:205] Tried to move back new op lower bound from 7333271960933289984 to 7333271960920854528. Current Snapshot: MvccSnapshot[applied={T|T < 7333271960933289984}]
I20260925 15:19:59.853192 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.835028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.8.43.190:38927)
0925 15:19:59.835064 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.835066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.837339 (+ 2273us) server_negotiation.cc:407] Connection header received
0925 15:19:59.837433 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.837434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.837443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.837471 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.840315 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.841046 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.842002 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.842298 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.842363 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.849337 (+ 6974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.849338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.849339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.849346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.851074 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.852187 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.852188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.852188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.852241 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.852472 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.852473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.852474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.852622 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.852713 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.852921 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.853053 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:19:59.856015 31109 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } attempt: 3
I20260925 15:19:59.858718 31222 catalog_manager.cc:1347] Generated new cluster ID: 1a5d143cee934c728d964c079723de31
I20260925 15:19:59.858821 31222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:19:59 dist-test-slave-c5d6 krb5kdc[30953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.43.189@KRBTEST.COM for kudu/127.8.43.188@KRBTEST.COM
I20260925 15:19:59.866598 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.856274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.43.188:36311 (local address 127.0.0.1:45586)
0925 15:19:59.856350 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 15:19:59.856360 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:19:59.856407 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:19:59.856670 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:19:59.856671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:19:59.856839 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:19:59.856880 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.856881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.857832 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.857833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.858596 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:19:59.858598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.859692 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.859693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:19:59.859756 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.860225 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:19:59.860230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:19:59.861658 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:19:59.865336 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.865337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.865337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.865557 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.865846 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:19:59.865846 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:19:59.865847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:19:59.865899 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:19:59.866302 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:19:59.866303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:19:59.866355 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:19:59.866409 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:19:59.866504 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:19:59.867018 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:19:59.856521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.8.43.188:36311)
0925 15:19:59.856556 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.856558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.856564 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:19:59.856583 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.856584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.856593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.856961 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.856991 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.857741 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.859330 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:19:59.859619 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:19:59.859676 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:19:59.861766 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:19:59.861767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:19:59.861768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:19:59.861774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:19:59.865197 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.865647 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.865648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.865649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.865743 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:19:59.865972 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:19:59.865972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:19:59.865973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:19:59.866175 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:19:59.866692 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0925 15:19:59.866763 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:19:59.866884 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:19:59.868460 31109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb: Bootstrap starting.
I20260925 15:19:59.869146 31109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb: Neither blocks nor log segments found. Creating new log.
I20260925 15:19:59.869455 31109 log.cc:824] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:19:59.873162 31222 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:19:59.873821 31222 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:19:59.877733 31222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc: Generated new TSK 0
I20260925 15:19:59.877882 31222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:19:59.884004 31109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb: No bootstrap required, opened a new log
I20260925 15:19:59.886201 31109 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } }
I20260925 15:19:59.886358 31109 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f29ad8dfa5449995eec57374e962bb, State: Initialized, Role: FOLLOWER
I20260925 15:19:59.886430 31109 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } }
I20260925 15:19:59.886745 31109 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:19:59.887014 31083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9f29ad8dfa5449995eec57374e962bb"
I20260925 15:19:59.887112 31083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:19:59.887974 31083 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fc4c6b562f34fc3a268573b891cf5bc in term 1.
I20260925 15:19:59.888360 31082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [term 1 FOLLOWER]: Refusing update from remote peer 7fc4c6b562f34fc3a268573b891cf5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:19:59.888657 31204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:19:59.893294 31249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:19:59.893347 31249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:19:59.893702 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.894459 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:59.894400 31236 mvcc.cc:205] Tried to move back new op lower bound from 7333271961094103040 to 7333271960920854528. Current Snapshot: MvccSnapshot[applied={T|T < 7333271960933289984}]
I20260925 15:19:59.894784 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc4c6b562f34fc3a268573b891cf5bc. Latest consensus state: current_term: 1 leader_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.894893 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:19:59.895000 31235 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:19:59.895102 31235 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:19:59.921244 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.919551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.8.43.189:33525)
0925 15:19:59.919597 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.919599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.920572 (+ 973us) server_negotiation.cc:407] Connection header received
0925 15:19:59.920672 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.920673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.920684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.920714 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.921166 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41172: BlockingRecv error: recv got EOF from 127.0.0.1:41172 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:19:59.921938 31188 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.919976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.8.43.188:36311)
0925 15:19:59.920030 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.920032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.921481 (+ 1449us) server_negotiation.cc:407] Connection header received
0925 15:19:59.921574 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.921575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.921585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.921612 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.921885 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45594: BlockingRecv error: recv got EOF from 127.0.0.1:45594 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 15:19:59.922607 31037 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.920221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.8.43.190:38927)
0925 15:19:59.920259 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.920261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.922146 (+ 1885us) server_negotiation.cc:407] Connection header received
0925 15:19:59.922233 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.922234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.922244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.922272 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.922547 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41396: BlockingRecv error: recv got EOF from 127.0.0.1:41396 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:19:59.924881 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.924138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.8.43.189:33525)
0925 15:19:59.924178 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.924180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.924388 (+ 208us) server_negotiation.cc:407] Connection header received
0925 15:19:59.924486 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.924487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.924504 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.924532 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.924826 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41182: BlockingRecv error: recv got EOF from 127.0.0.1:41182 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 15:19:59.931063 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.927544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.8.43.189:33525)
0925 15:19:59.927575 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.927577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.930564 (+ 2987us) server_negotiation.cc:407] Connection header received
0925 15:19:59.930664 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.930665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.930675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.930704 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.931021 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41184: BlockingRecv error: recv got EOF from 127.0.0.1:41184 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 15:19:59.934875 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.934097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.8.43.189:33525)
0925 15:19:59.934129 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.934131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.934406 (+ 275us) server_negotiation.cc:407] Connection header received
0925 15:19:59.934501 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.934502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.934512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.934541 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.934836 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41194: BlockingRecv error: recv got EOF from 127.0.0.1:41194 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 15:19:59.940797 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.939943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.8.43.189:33525)
0925 15:19:59.939976 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.939978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.940272 (+ 294us) server_negotiation.cc:407] Connection header received
0925 15:19:59.940415 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.940416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.940426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.940454 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.940758 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41210: BlockingRecv error: recv got EOF from 127.0.0.1:41210 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 15:19:59.951076 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.949946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.8.43.189:33525)
0925 15:19:59.950004 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.950006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.950466 (+ 460us) server_negotiation.cc:407] Connection header received
0925 15:19:59.950570 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.950571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.950583 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.950612 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.951014 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41216: BlockingRecv error: recv got EOF from 127.0.0.1:41216 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:19:59.969312 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:19:59.968407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.8.43.189:33525)
0925 15:19:59.968451 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:19:59.968453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:19:59.968711 (+ 258us) server_negotiation.cc:407] Connection header received
0925 15:19:59.968815 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:19:59.968816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:19:59.968828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:19:59.968858 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:19:59.969255 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41228: BlockingRecv error: recv got EOF from 127.0.0.1:41228 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:20:00.003616 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:00.002787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.8.43.189:33525)
0925 15:20:00.002837 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.002840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.002982 (+ 142us) server_negotiation.cc:407] Connection header received
0925 15:20:00.003065 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.003067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.003079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.003154 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:00.003545 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41230: BlockingRecv error: recv got EOF from 127.0.0.1:41230 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:20:00.069512 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:00.068757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.8.43.189:33525)
0925 15:20:00.068813 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.068815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.068929 (+ 114us) server_negotiation.cc:407] Connection header received
0925 15:20:00.069014 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.069015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.069027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.069111 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:00.069428 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41234: BlockingRecv error: recv got EOF from 127.0.0.1:41234 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 15:20:00.199534 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:00.198786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.8.43.189:33525)
0925 15:20:00.198899 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.198901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.198916 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:20:00.199023 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.199024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.199035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.199088 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:00.199469 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41236: BlockingRecv error: recv got EOF from 127.0.0.1:41236 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:20:00.275974 31216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fc4c6b562f34fc3a268573b891cf5bc" member_type: VOTER last_known_addr { host: "127.8.43.190" port: 38927 } } peers { permanent_uuid: "a9f29ad8dfa5449995eec57374e962bb" member_type: VOTER last_known_addr { host: "127.8.43.189" port: 33525 } } peers { permanent_uuid: "2c43db0f41fe423592cd6b2006e99d82" member_type: VOTER last_known_addr { host: "127.8.43.188" port: 36311 } } }
I20260925 15:20:00.276175 31216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc4c6b562f34fc3a268573b891cf5bc [sys.catalog]: This master's current role is: LEADER
W20260925 15:20:00.457369 31113 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:00.456695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.8.43.189:33525)
0925 15:20:00.456790 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.456792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.456806 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:20:00.456840 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.456841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.456862 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.456933 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:00.457321 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41246: BlockingRecv error: recv got EOF from 127.0.0.1:41246 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:20:00.808844 31201 catalog_manager.cc:1259] Loaded cluster ID: 1a5d143cee934c728d964c079723de31
I20260925 15:20:00.808940 31201 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82: loading cluster ID for follower catalog manager: success
I20260925 15:20:00.809857 31201 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82: acquiring CA information for follower catalog manager: success
I20260925 15:20:00.810168 31201 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c43db0f41fe423592cd6b2006e99d82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:20:00.893754 31249 catalog_manager.cc:1259] Loaded cluster ID: 1a5d143cee934c728d964c079723de31
I20260925 15:20:00.893831 31249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb: loading cluster ID for follower catalog manager: success
I20260925 15:20:00.894657 31249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb: acquiring CA information for follower catalog manager: success
I20260925 15:20:00.894919 31249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9f29ad8dfa5449995eec57374e962bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:20:00.973658 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:00.970892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.8.43.189:33525)
0925 15:20:00.971031 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.971033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.971090 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:20:00.971174 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.971175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.971259 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.971351 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:00.971524 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.972083 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.972864 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.973068 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.973143 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.973312 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:00.973427 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:00.973516 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:00.973541 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 15:20:00.973561 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":56,"threads_started":1}
I20260925 15:20:00.977772 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:00.975428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.8.43.188:36311)
0925 15:20:00.975554 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.975556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.975564 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:00.975628 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.975629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.975661 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.975727 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:00.975812 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.976270 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.977048 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.977223 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.977270 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.977384 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:00.977483 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:00.977536 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:00.977623 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:20:00.977639 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260925 15:20:00.982412 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:00.979441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.8.43.188:36311)
0925 15:20:00.979524 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.979526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.979535 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:00.979630 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.979632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.979646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.979708 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:00.979828 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.980564 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.981578 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.981857 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.981913 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.982062 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:00.982094 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:00.982141 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:00.982241 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:20:00.982261 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:20:00.982441 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:00.979410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.8.43.189:33525)
0925 15:20:00.979483 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.979485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.979953 (+ 468us) server_negotiation.cc:407] Connection header received
0925 15:20:00.980052 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.980052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.980063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.980089 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:00.980253 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.980855 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.981626 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.981918 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.981991 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.982151 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:00.982181 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:00.982229 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:00.982343 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:20:00.982370 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:20:00.982678 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:00.979521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.8.43.190:38927)
0925 15:20:00.979622 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:00.979624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:00.979632 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:00.979680 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:00.979681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:00.979733 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:00.979814 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:00.979918 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.980401 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.981486 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:00.981769 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:00.981847 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:00.982041 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:00.982160 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:00.982228 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:00.982396 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 15:20:00.982428 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":41,"threads_started":1}
I20260925 15:20:00.993012 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 30966
I20260925 15:20:00.998246 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 31041
I20260925 15:20:01.003767 8366 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcJTRKe/build/release/bin/kudu with pid 31116
2026-09-25T15:20:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2802 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5598 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (294406 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS